[jira] [Created] (HIVE-12344) Wrong types inferred for SemiJoin generation in CBO
Jesus Camacho Rodriguez created HIVE-12344:
--
Summary: Wrong types inferred for SemiJoin generation in CBO
Key: HIVE-12344
URL: https://issues.apache.org/jira/browse/HIVE-12344
Project: Hive
Issue Type: Bug
Components: CBO
Affects Versions: 1.3.0, 2.0.0
Reporter: Jesus Camacho Rodriguez
Assignee: Jesus Camacho Rodriguez
The method {{projectNonColumnEquiConditions}} in HiveCalciteUtil will assign
the type wrongly for newly created conditions. The problem is in this block:
{noformat}
RexNode cond = rexBuilder.makeCall(SqlStdOperatorTable.EQUALS,
rexBuilder.makeInputRef(newLeftFields.get(i).getType(), newLeftOffset
+ i),
rexBuilder.makeInputRef(newLeftFields.get(i).getType(),
newRightOffset + i));
{noformat}
It looks like a code copy-paste mistake. In addition, index {{i}} is incorrect,
as newLeftFields contains all the fields, not only the ones of the new
condition.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12345) Followup for HIVE-9013 : Hidden commands still visible through beeline
Sushanth Sowmyan created HIVE-12345: --- Summary: Followup for HIVE-9013 : Hidden commands still visible through beeline Key: HIVE-12345 URL: https://issues.apache.org/jira/browse/HIVE-12345 Project: Hive Issue Type: Bug Reporter: Sushanth Sowmyan Assignee: Sushanth Sowmyan HIVE-9013 introduced the ability to hide certain conf variables when output through the "set" command. However, there still exists one further bug in it that causes these variables to still be visible through beeline connecting to HS2, wherein HS2 exposes hidden variables such as the HS2's metastore password when "set" is run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 39973: HIVE-12346: Internally used variables in HiveConf should not be settable via command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39973/#review105242 --- Ship it! Ship It! - Xuefu Zhang On Nov. 5, 2015, 1:59 p.m., Chaoyu Tang wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/39973/ > --- > > (Updated Nov. 5, 2015, 1:59 p.m.) > > > Review request for hive. > > > Bugs: HIVE-12346 > https://issues.apache.org/jira/browse/HIVE-12346 > > > Repository: hive-git > > > Description > --- > > Internally used variables are put in a HiveConf > HIVE_CONF_INTERNAL_VARIABLE_LIST, which is a part of restricted list. > > > Diffs > - > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ab73ad > ql/src/test/queries/clientnegative/set_hiveconf_internal_variable0.q > PRE-CREATION > ql/src/test/queries/clientnegative/set_hiveconf_internal_variable1.q > PRE-CREATION > ql/src/test/results/clientnegative/set_hiveconf_internal_variable0.q.out > PRE-CREATION > ql/src/test/results/clientnegative/set_hiveconf_internal_variable1.q.out > PRE-CREATION > > Diff: https://reviews.apache.org/r/39973/diff/ > > > Testing > --- > > New qtests passed > Running precommit build > > > Thanks, > > Chaoyu Tang > >
[jira] [Created] (HIVE-12349) NPE in ORC SARG for IS NULL queries on Timestamp and Date columns
Prasanth Jayachandran created HIVE-12349:
Summary: NPE in ORC SARG for IS NULL queries on Timestamp and Date
columns
Key: HIVE-12349
URL: https://issues.apache.org/jira/browse/HIVE-12349
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.1, 1.3.0, 2.0.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
IS NULL queries can trigger an NPE for timestamp and date columns. All column
values per row group or stripe should be NULL to trigger this case. Following
is the exception stack trace
{code}
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$TimestampStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:795)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getMin(RecordReaderImpl.java:2343)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:2366)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:2564)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2627)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:3060)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:3102)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.(RecordReaderImpl.java:288)
at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:534)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.(OrcRawRecordMerger.java:183)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.(OrcRawRecordMerger.java:226)
at
org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.(OrcRawRecordMerger.java:437)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1141)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1039)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:246)
... 26 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12348) String comparison optimization
Teddy Choi created HIVE-12348: - Summary: String comparison optimization Key: HIVE-12348 URL: https://issues.apache.org/jira/browse/HIVE-12348 Project: Hive Issue Type: Sub-task Reporter: Teddy Choi Assignee: Teddy Choi Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12347) Different tables' external partitions can point to the same location without any warning or error
Anne Yu created HIVE-12347:
--
Summary: Different tables' external partitions can point to the
same location without any warning or error
Key: HIVE-12347
URL: https://issues.apache.org/jira/browse/HIVE-12347
Project: Hive
Issue Type: Bug
Reporter: Anne Yu
{code}
CREATE TABLE testloc1 (test STRING) PARTITIONED BY(day INT, hour INT, min INT);
CREATE TABLE testloc2 (test STRING) PARTITIONED BY(day INT, hour INT, min INT);
ALTER TABLE testloc1 add partition (day="01", hour="01", min="01") location
'/hive/pipeline/data/01/01/01’;
ALTER TABLE testloc2 add partition (day="01", hour="01", min="01") location
'/hive/pipeline/data/01/01/01’;
{code}
The issue is if overwrite data into testloc1 then testloc2, testloc1 data will
be wiped out. For example,
{code}
insert overwrite table testloc1 partition (day="01", hour="01", min="01")
select "test1" from test_tb;
insert overwrite table testloc2 partition (day="01", hour="01", min="01")
select "test2" from test_tb;
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-12357) Allow user to set tez job name
Gunther Hagleitner created HIVE-12357: - Summary: Allow user to set tez job name Key: HIVE-12357 URL: https://issues.apache.org/jira/browse/HIVE-12357 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Need something like mapred.job.name. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12356) Capture if a rule mutated the plan.
Laljo John Pullokkaran created HIVE-12356: - Summary: Capture if a rule mutated the plan. Key: HIVE-12356 URL: https://issues.apache.org/jira/browse/HIVE-12356 Project: Hive Issue Type: Bug Affects Versions: 1.2.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Currently Hive Optimizer doesn't capture if a rule mutated the plan. This info could be useful in: 1. determining if a subsequent optimization rule needs to be run or not (Ex if Constant propagation didn't mutate plan, then don't run subsequent PPD) 2. Explain can contain info about which optimizations are applied effectively on the query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12355) Keep Obj Inspectors in Sync with RowSchema
Laljo John Pullokkaran created HIVE-12355: - Summary: Keep Obj Inspectors in Sync with RowSchema Key: HIVE-12355 URL: https://issues.apache.org/jira/browse/HIVE-12355 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 1.2.1, 1.1.0, 1.0.0 Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Currently Not all operators match their Output Obj inspectors to Row schema. Many times OutputObjectInspectors may be more than needed. This causes problems especially with union. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12353) When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not.
Eugene Koifman created HIVE-12353: - Summary: When Compactor fails it calls CompactionTxnHandler.markedCleaned(). it should not. Key: HIVE-12353 URL: https://issues.apache.org/jira/browse/HIVE-12353 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman One of the things that this method does is delete entries from TXN_COMPONENTS for partition that it was trying to compact. This causes Aborted transactions in TXNS to become empty according to CompactionTxnHandler.cleanEmptyAbortedTxns() which means they can now be delete. We should extend COMPACTION_QUEUE state with 'f' and 's' (failed, success) states. We should also not delete then entry from markedCleaned() We'll have separate process that cleans 'f' and 's' records after X minutes (or after > N records for a given partition exist). This allows SHOW COMPACTIONS to show some history info and how many times compaction failed on a given partition (subject to retention interval) so that we don't have to call markCleaned() on Compactor failures at the same time preventing Compactor to constantly getting stuck on the same bad partition/table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 39973: HIVE-12346: Internally used variables in HiveConf should not be settable via command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39973/ --- (Updated Nov. 5, 2015, 11:27 p.m.) Review request for hive. Changes --- Revise patch based on review. Bugs: HIVE-12346 https://issues.apache.org/jira/browse/HIVE-12346 Repository: hive-git Description --- Internally used variables are put in a HiveConf HIVE_CONF_INTERNAL_VARIABLE_LIST, which is a part of restricted list. Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ab73ad ql/src/test/queries/clientnegative/set_hiveconf_internal_variable0.q PRE-CREATION ql/src/test/queries/clientnegative/set_hiveconf_internal_variable1.q PRE-CREATION ql/src/test/results/clientnegative/set_hiveconf_internal_variable0.q.out PRE-CREATION ql/src/test/results/clientnegative/set_hiveconf_internal_variable1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/39973/diff/ Testing --- New qtests passed Running precommit build Thanks, Chaoyu Tang
[jira] [Created] (HIVE-12352) CompactionTxnHandler.markCleaned() may delete too much
Eugene Koifman created HIVE-12352: - Summary: CompactionTxnHandler.markCleaned() may delete too much Key: HIVE-12352 URL: https://issues.apache.org/jira/browse/HIVE-12352 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 1.0.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Worker will start with DB in state X (wrt this partition). while it's working more txns will happen, against partition it's compacting. then this will delete state up to X and since then. There may be new delta files created between compaction starting and cleaning. These will not be compacted until more transactions happen. So this ideally should only delete up to TXN_ID that was compacted (i.e. HWM in Worker?) Then this can also run at READ_COMMITTED. So this means we'd want to store HWM in COMPACTION_QUEUE when Worker picks up the job. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 39960: HIVE-12331
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39960/ --- (Updated Nov. 5, 2015, 8:50 p.m.) Review request for hive and Gopal V. Changes --- Fixed test failures Bugs: HIVE-12331 https://issues.apache.org/jira/browse/HIVE-12331 Repository: hive-git Description --- Remove hive.enforce.bucketing & hive.enforce.sorting Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ab73ad ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java 39a881a ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 02adf0c ql/src/java/org/apache/hadoop/hive/ql/optimizer/SortedDynPartitionOptimizer.java e2a0eae ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d2c3a7c ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java e13e6eb ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 5aa2500 ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 44ad8b0 ql/src/test/queries/clientnegative/acid_overwrite.q 2e57a3c ql/src/test/queries/clientnegative/archive_corrupt.q 130b37b ql/src/test/queries/clientnegative/authorization_delete_nodeletepriv.q f2de306 ql/src/test/queries/clientnegative/authorization_update_noupdatepriv.q c00c0eb ql/src/test/queries/clientnegative/delete_non_acid_table.q 6ae82ff ql/src/test/queries/clientnegative/delete_not_bucketed.q 80dffea ql/src/test/queries/clientnegative/delete_sorted.q fd8d579 ql/src/test/queries/clientnegative/insert_sorted.q 18c942a ql/src/test/queries/clientnegative/insert_values_sorted.q 260e2fb ql/src/test/queries/clientnegative/merge_negative_3.q 6bc645e ql/src/test/queries/clientnegative/smb_bucketmapjoin.q 880323c ql/src/test/queries/clientnegative/smb_mapjoin_14.q 54bfba0 ql/src/test/queries/clientnegative/sortmerge_mapjoin_mismatch_1.q 7d11f45 ql/src/test/queries/clientnegative/update_bucket_col.q 515e024 ql/src/test/queries/clientnegative/update_no_such_table.q 07239cf ql/src/test/queries/clientnegative/update_non_acid_table.q dd0b01e ql/src/test/queries/clientnegative/update_not_bucketed.q 8512fa7 ql/src/test/queries/clientnegative/update_partition_col.q e9c60cc ql/src/test/queries/clientnegative/update_sorted.q 917c3b5 ql/src/test/queries/clientpositive/acid_join.q 2e6aeae ql/src/test/queries/clientpositive/acid_vectorization.q 4b11412 ql/src/test/queries/clientpositive/acid_vectorization_partition.q 00449bb ql/src/test/queries/clientpositive/acid_vectorization_project.q a44b57a ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table.q 627fcc1 ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table2.q 2f26de8 ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table2_h23.q 2c2e184 ql/src/test/queries/clientpositive/alter_numbuckets_partitioned_table_h23.q 439f351 ql/src/test/queries/clientpositive/archive_excludeHadoop20.q 316276a ql/src/test/queries/clientpositive/archive_multi.q 1004aca ql/src/test/queries/clientpositive/authorization_delete.q d96e6ab ql/src/test/queries/clientpositive/authorization_delete_own_table.q 7abdc12 ql/src/test/queries/clientpositive/authorization_update.q da1054e ql/src/test/queries/clientpositive/authorization_update_own_table.q ace1ce2 ql/src/test/queries/clientpositive/auto_smb_mapjoin_14.q 699777e ql/src/test/queries/clientpositive/auto_sortmerge_join_10.q c07dd23 ql/src/test/queries/clientpositive/auto_sortmerge_join_13.q f35fec1 ql/src/test/queries/clientpositive/auto_sortmerge_join_14.q eabeff0 ql/src/test/queries/clientpositive/auto_sortmerge_join_15.q a553d93 ql/src/test/queries/clientpositive/auto_sortmerge_join_16.q cb244cf ql/src/test/queries/clientpositive/auto_sortmerge_join_6.q 0ddf378 ql/src/test/queries/clientpositive/auto_sortmerge_join_9.q 9eb85d3 ql/src/test/queries/clientpositive/bucket1.q 0154b4e ql/src/test/queries/clientpositive/bucket2.q ecd7e53 ql/src/test/queries/clientpositive/bucket3.q 7b7a9c3 ql/src/test/queries/clientpositive/bucket4.q 1b49c7a ql/src/test/queries/clientpositive/bucket5.q 877f8a5 ql/src/test/queries/clientpositive/bucket6.q fb55787 ql/src/test/queries/clientpositive/bucket_many.q 1f0b795 ql/src/test/queries/clientpositive/bucket_map_join_1.q 6bdb09e ql/src/test/queries/clientpositive/bucket_map_join_2.q 07f6d15 ql/src/test/queries/clientpositive/bucket_map_join_spark4.q 1ca20e4 ql/src/test/queries/clientpositive/bucket_map_join_tez1.q 8546e78 ql/src/test/queries/clientpositive/bucket_map_join_tez2.q 2f968bd ql/src/test/queries/clientpositive/bucket_num_reducers.q 37ae6cc ql/src/test/queries/clientpositive/bucket_num_reducers2.q 8c64d60 ql/src/test/queries/clientpositive/bucketmapjoin13.q f01c43e
[jira] [Created] (HIVE-12350) Boolean bitwise operation optimization for SIMD
Teddy Choi created HIVE-12350: - Summary: Boolean bitwise operation optimization for SIMD Key: HIVE-12350 URL: https://issues.apache.org/jira/browse/HIVE-12350 Project: Hive Issue Type: Sub-task Reporter: Teddy Choi Assignee: Teddy Choi Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12351) add mssql tables for authorization in pre-upgrade script
Thejas M Nair created HIVE-12351: Summary: add mssql tables for authorization in pre-upgrade script Key: HIVE-12351 URL: https://issues.apache.org/jira/browse/HIVE-12351 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.14.0 Reporter: Thejas M Nair Assignee: Thejas M Nair With schematool becoming increasingly the tool of choice for upgrades, datanucleus.autoCreateSchema is often turned off for newer versions of hive. However, this can be a problem if old schema was created using DataNucleus autocreate-schema, it might not have created the tables related to authorization if authorization functionality was not being used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12354) MapJoin with double keys is slow on MR
Sergey Shelukhin created HIVE-12354: --- Summary: MapJoin with double keys is slow on MR Key: HIVE-12354 URL: https://issues.apache.org/jira/browse/HIVE-12354 Project: Hive Issue Type: Bug Reporter: Sergey Shelukhin Assignee: Sergey Shelukhin Double keys are also a common type when comparing numbers with strings, and such, so it happens more often than one would expect. This is due to HADOOP-12217 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12358) Categorize vectorization benchmarks into arithmetic, comparison, logic
Teddy Choi created HIVE-12358: - Summary: Categorize vectorization benchmarks into arithmetic, comparison, logic Key: HIVE-12358 URL: https://issues.apache.org/jira/browse/HIVE-12358 Project: Hive Issue Type: Sub-task Reporter: Teddy Choi Assignee: Teddy Choi There are 30+ vectorization benchmarks in VectorizationBench.java file with 500+ lines. They need to be grouped by categories into arithmetic, logic, comparison for ease of management. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-12346) Internally used variables in HiveConf should not be settable via command
Chaoyu Tang created HIVE-12346: -- Summary: Internally used variables in HiveConf should not be settable via command Key: HIVE-12346 URL: https://issues.apache.org/jira/browse/HIVE-12346 Project: Hive Issue Type: Bug Components: Configuration Affects Versions: 1.2.1 Reporter: Chaoyu Tang Assignee: Chaoyu Tang Some HiveConf variables such as hive.added.jars.path are only for internal use and should not be settable via set command. We saw a lot of cases that users mistakenly set these variables using set command despite some of them have been documented as "internal parameter" in Hive. The command usually succeeds but it sometimes does not effect, which causes some confusions. For example, the hive.added.jars.path can be set via set command but it is sometimes overridden by session resource jars during runtime. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 39973: HIVE-12346: Internally used variables in HiveConf should not be settable via command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/39973/ --- Review request for hive. Bugs: HIVE-12346 https://issues.apache.org/jira/browse/HIVE-12346 Repository: hive-git Description --- Internally used variables are put in a HiveConf HIVE_CONF_INTERNAL_VARIABLE_LIST, which is a part of restricted list. Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 3ab73ad ql/src/test/queries/clientnegative/set_hiveconf_internal_variable0.q PRE-CREATION ql/src/test/queries/clientnegative/set_hiveconf_internal_variable1.q PRE-CREATION ql/src/test/results/clientnegative/set_hiveconf_internal_variable0.q.out PRE-CREATION ql/src/test/results/clientnegative/set_hiveconf_internal_variable1.q.out PRE-CREATION Diff: https://reviews.apache.org/r/39973/diff/ Testing --- New qtests passed Running precommit build Thanks, Chaoyu Tang
