[jira] [Created] (HIVE-20688) Update Committer List
Janaki Lahorani created HIVE-20688: -- Summary: Update Committer List Key: HIVE-20688 URL: https://issues.apache.org/jira/browse/HIVE-20688 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani Please update committer list: Name: Janaki Lahorani Apache ID: janaki Organization: Cloudera -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20570) Union ALL with hive.optimize.union.remove=true has incorrect plan
Janaki Lahorani created HIVE-20570:
--
Summary: Union ALL with hive.optimize.union.remove=true has
incorrect plan
Key: HIVE-20570
URL: https://issues.apache.org/jira/browse/HIVE-20570
Project: Hive
Issue Type: Bug
Reporter: Janaki Lahorani
When hive.optimize.union.remove=true and a select query is run with group by,
the final fetch is waiting only for one of the branches and not both.
Test Case:
{code}
create table if not exists test_table(column1 string, column2 int);
insert into test_table values('a',1),('b',2);
set hive.optimize.union.remove=true;
set mapred.input.dir.recursive=true;
explain
select column1 from test_table group by column1
union all
select column1 from test_table group by column1;
{code}
In the below the two stages correspond to the two parts of union all. But the
final fetch operator (Stage 0) only depends on one of the stages, but it should
depend on both.
Plan:
{code}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 is a root stage
* Stage-0 depends on stages: Stage-1*
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: test_table
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
stats: NONE
Select Operator
expressions: column1 (type: string)
outputColumnNames: column1
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
stats: NONE
Group By Operator
keys: column1 (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
Column stats: NONE
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-2
Map Reduce
Map Operator Tree:
TableScan
alias: test_table
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
stats: NONE
Select Operator
expressions: column1 (type: string)
outputColumnNames: column1
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE Column
stats: NONE
Group By Operator
keys: column1 (type: string)
mode: hash
outputColumnNames: _col0
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
Column stats: NONE
Reduce Output Operator
key expressions: _col0 (type: string)
sort order: +
Map-reduce partition columns: _col0 (type: string)
Statistics: Num rows: 2 Data size: 6 Basic stats: COMPLETE
Column stats: NONE
Execution mode: vectorized
Reduce Operator Tree:
Group By Operator
keys: KEY._col0 (type: string)
mode: mergepartial
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
stats: NONE
File Output Operator
compressed: false
Statistics: Num rows: 1 Data size: 3 Basic stats: COMPLETE Column
stats: NONE
table:
input format: org.apache.hadoop.mapred.SequenceFileInputFormat
output format:
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
ListSink
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-20562) Intermittent test failures from Druid tests
Janaki Lahorani created HIVE-20562: -- Summary: Intermittent test failures from Druid tests Key: HIVE-20562 URL: https://issues.apache.org/jira/browse/HIVE-20562 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Druid tests are failing intermittently in Hive Pre-commit jobs. The typical failures include: org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_alter] (batchId=193) org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert] (batchId=193) The test log shows the following: Exception: org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (java.net.ConnectException : Error connecting to server localhost on port 60,000 with message Connection refused.) org.apache.hadoop.hive.ql.metadata.HiveException: org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create PoolableConnectionFactory (java.net.ConnectException : Error connecting to server localhost on port 60,000 with message Connection refused.) at org.apache.hadoop.hive.ql.metadata.Hive.dropTable(Hive.java:1077) at org.apache.hadoop.hive.ql.QTestUtil.clearTablesCreatedDuringTests(QTestUtil.java:958) at org.apache.hadoop.hive.ql.QTestUtil.clearTestSideEffects(QTestUtil.java:1039) at org.apache.hadoop.hive.cli.control.CoreCliDriver$5.invokeInternal(CoreCliDriver.java:135) at org.apache.hadoop.hive.cli.control.CoreCliDriver$5.invokeInternal(CoreCliDriver.java:131) at org.apache.hadoop.hive.util.ElapsedTimeLoggingWrapper.invoke(ElapsedTimeLoggingWrapper.java:33) at org.apache.hadoop.hive.cli.control.CoreCliDriver.tearDown(CoreCliDriver.java:138) at org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:94) The following search shows many Hive Jiras with patches where Druid tests are failing. https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20text%20~%20druidmini%20ORDER%20BY%20key%20DESC -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20527) Intern table descriptors from spark task
Janaki Lahorani created HIVE-20527: -- Summary: Intern table descriptors from spark task Key: HIVE-20527 URL: https://issues.apache.org/jira/browse/HIVE-20527 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani Table descriptors from MR tasks and Tez tasks are interned. This fix is to intern table desc from spark tasks as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20526) Add test case for HIVE-20489
Janaki Lahorani created HIVE-20526: -- Summary: Add test case for HIVE-20489 Key: HIVE-20526 URL: https://issues.apache.org/jira/browse/HIVE-20526 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani Add a test case for the issue discussed in HIVE-20489. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20489) Explain plan of query hangs
Janaki Lahorani created HIVE-20489: -- Summary: Explain plan of query hangs Key: HIVE-20489 URL: https://issues.apache.org/jira/browse/HIVE-20489 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Explain on a query that joins 47 views, in effect around 94 joins after view expansion seems to take forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-20437) Handle schema evolution from float, double and decimal
Janaki Lahorani created HIVE-20437: -- Summary: Handle schema evolution from float, double and decimal Key: HIVE-20437 URL: https://issues.apache.org/jira/browse/HIVE-20437 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani When data created as float, double or decimal in parquet format is read back using some other type, errors are seen. Parquet should behave just like any other format. If the value is valid for the new type, data is retuned otherwise null has to be returned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19940) Push predicates with deterministic UDFs with RBO
Janaki Lahorani created HIVE-19940:
--
Summary: Push predicates with deterministic UDFs with RBO
Key: HIVE-19940
URL: https://issues.apache.org/jira/browse/HIVE-19940
Project: Hive
Issue Type: Bug
Reporter: Janaki Lahorani
Assignee: Janaki Lahorani
With RBO, predicates with any UDF doesn't get pushed down. It makes sense to
not pushdown the predicates with non-deterministic function as the meaning of
the query changes after the predicate is resolved to use the function. But
pushing a deterministic function is beneficial.
Test Case:
{code}
set hive.cbo.enable=false;
CREATE TABLE `testb`(
`cola` string COMMENT '',
`colb` string COMMENT '',
`colc` string COMMENT '')
PARTITIONED BY (
`part1` string,
`part2` string,
`part3` string)
STORED AS AVRO;
CREATE TABLE `testa`(
`col1` string COMMENT '',
`col2` string COMMENT '',
`col3` string COMMENT '',
`col4` string COMMENT '',
`col5` string COMMENT '')
PARTITIONED BY (
`part1` string,
`part2` string,
`part3` string)
STORED AS AVRO;
insert into testA partition (part1='US', part2='ABC', part3='123')
values ('12.34', '100', '200', '300', 'abc'),
('12.341', '1001', '2001', '3001', 'abcd');
insert into testA partition (part1='UK', part2='DEF', part3='123')
values ('12.34', '100', '200', '300', 'abc'),
('12.341', '1001', '2001', '3001', 'abcd');
insert into testA partition (part1='US', part2='DEF', part3='200')
values ('12.34', '100', '200', '300', 'abc'),
('12.341', '1001', '2001', '3001', 'abcd');
insert into testA partition (part1='CA', part2='ABC', part3='300')
values ('12.34', '100', '200', '300', 'abc'),
('12.341', '1001', '2001', '3001', 'abcd');
insert into testB partition (part1='CA', part2='ABC', part3='300')
values ('600', '700', 'abc'), ('601', '701', 'abcd');
insert into testB partition (part1='CA', part2='ABC', part3='400')
values ( '600', '700', 'abc'), ( '601', '701', 'abcd');
insert into testB partition (part1='UK', part2='PQR', part3='500')
values ('600', '700', 'abc'), ('601', '701', 'abcd');
insert into testB partition (part1='US', part2='DEF', part3='200')
values ( '600', '700', 'abc'), ('601', '701', 'abcd');
insert into testB partition (part1='US', part2='PQR', part3='123')
values ( '600', '700', 'abc'), ('601', '701', 'abcd');
-- views with deterministic functions
create view viewDeterministicUDFA partitioned on (vpart1, vpart2, vpart3) as
select
cast(col1 as decimal(38,18)) as vcol1,
cast(col2 as decimal(38,18)) as vcol2,
cast(col3 as decimal(38,18)) as vcol3,
cast(col4 as decimal(38,18)) as vcol4,
cast(col5 as char(10)) as vcol5,
cast(part1 as char(2)) as vpart1,
cast(part2 as char(3)) as vpart2,
cast(part3 as char(3)) as vpart3
from testa
where part1 in ('US', 'CA');
create view viewDeterministicUDFB partitioned on (vpart1, vpart2, vpart3) as
select
cast(cola as decimal(38,18)) as vcolA,
cast(colb as decimal(38,18)) as vcolB,
cast(colc as char(10)) as vcolC,
cast(part1 as char(2)) as vpart1,
cast(part2 as char(3)) as vpart2,
cast(part3 as char(3)) as vpart3
from testb
where part1 in ('US', 'CA');
explain
select vcol1, vcol2, vcol3, vcola, vcolb
from viewDeterministicUDFA a inner join viewDeterministicUDFB b
on a.vpart1 = b.vpart1
and a.vpart2 = b.vpart2
and a.vpart3 = b.vpart3
and a.vpart1 = 'US'
and a.vpart2 = 'DEF'
and a.vpart3 = '200';
{code}
Plan where the CAST is not pushed down.
{code}
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: testa
filterExpr: (part1) IN ('US', 'CA') (type: boolean)
Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
Column stats: NONE
Select Operator
expressions: CAST( col1 AS decimal(38,18)) (type:
decimal(38,18)), CAST( col2 AS decimal(38,18)) (type: decimal(38,18)), CAST(
col3 AS decimal(38,18)) (type: decimal(38,18)), CAST( part1 AS CHAR(2)) (type:
char(2)), CAST( part2 AS CHAR(3)) (type: char(3)), CAST( part3 AS CHAR(3))
(type: char(3))
outputColumnNames: _col0, _col1, _col2, _col5, _col6, _col7
Statistics: Num rows: 6 Data size: 13740 Basic stats: COMPLETE
Column stats: NONE
Filter Operator
predicate: ((_col5 = 'US') and (_col6 = 'DEF') and (_col7 =
'200')) (type: boolean)
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
Reduce Output Operator
key expressions: 'US' (type: char(2)), 'DEF' (type: char(3)),
'200' (type: char(3))
sort order: +++
Map-reduce partition columns: 'US' (type: char(2)), 'DEF'
(type: char(3)), '200' (type: char(3))
Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE
Column stats: NONE
value expressions: _col0 (type: decimal(38,18
[jira] [Created] (HIVE-19889) Wrong results due to PPD of non deterministic functions with CBO
Janaki Lahorani created HIVE-19889: -- Summary: Wrong results due to PPD of non deterministic functions with CBO Key: HIVE-19889 URL: https://issues.apache.org/jira/browse/HIVE-19889 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani The following query can give wrong results when CBO is on: select * from ( select part1,randum123 from (SELECT *, cast(rand() as double) AS randum123 FROM testA where part1='CA' and part2 = 'ABC') a where randum123 <= 0.5) s where s.randum123 > 0.25 limit 20; The plan of the query is as follows: STAGE PLANS: Stage: Stage-1 Map Reduce Map Operator Tree: TableScan alias: testa Statistics: Num rows: 2 Data size: 4580 Basic stats: COMPLETE Column stats: NONE Filter Operator predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean) Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE Column stats: NONE Select Operator expressions: 'CA' (type: string), rand() (type: double) outputColumnNames: _col0, _col1 Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE Column stats: NONE Limit Number of rows: 20 Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE Column stats: NONE File Output Operator compressed: false Statistics: Num rows: 1 Data size: 2290 Basic stats: COMPLETE Column stats: NONE table: input format: org.apache.hadoop.mapred.SequenceFileInputFormat output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe Stage: Stage-0 Fetch Operator limit: 20 Processor Tree: ListSink The relevant part in the plan is the filter: Filter Operator predicate: ((rand() <= 0.5D) and (rand() > 0.25D)) (type: boolean) The predicates s.randum123 > 0.25 and s.randum123 > 0.25 were pushed down. And randum123 was resolved to rand(). This is bad because it will result in invocation of rand() two times and rand() UDF is non-deterministic. Both the rand calls can generate values that can satisfy the predicates independently, but not together, whereas the original intention of the query is to give results when rand falls between 0.25 and 0.5. A sample result: CA 0.9191984370369802 CA 0.397933021566812 where the condition was not satisfied. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19662) Upgrade Avro to 1.8.2
Janaki Lahorani created HIVE-19662: -- Summary: Upgrade Avro to 1.8.2 Key: HIVE-19662 URL: https://issues.apache.org/jira/browse/HIVE-19662 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani Upgrade Avro to 1.8.2 and use the version from root pom where Avro is referenced. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19421) Upgrade versions of Jetty and Jackson
Janaki Lahorani created HIVE-19421: -- Summary: Upgrade versions of Jetty and Jackson Key: HIVE-19421 URL: https://issues.apache.org/jira/browse/HIVE-19421 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Move Jackson up to 2.9.5 Move Jetty up to 9.3.20.v20170531 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19317) Handle schema evolution from int like types to decimal
Janaki Lahorani created HIVE-19317: -- Summary: Handle schema evolution from int like types to decimal Key: HIVE-19317 URL: https://issues.apache.org/jira/browse/HIVE-19317 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani If int like type is changed to decimal on parquet data, select results in errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19228) Remove commons-httpclient 3.x usage
Janaki Lahorani created HIVE-19228: -- Summary: Remove commons-httpclient 3.x usage Key: HIVE-19228 URL: https://issues.apache.org/jira/browse/HIVE-19228 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani Commons-httpclient is not supported well anymore. Remove dependency and move to Apache HTTP client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19182) Code clean up of HiveMetaStoreChecker.java
Janaki Lahorani created HIVE-19182: -- Summary: Code clean up of HiveMetaStoreChecker.java Key: HIVE-19182 URL: https://issues.apache.org/jira/browse/HIVE-19182 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani HiveMetaStoreChecker.java has code like: result.getTablesNotOnFs().add(table.getTableName()); result.getPartitionsNotOnFs().add(pr); result.getPartitionsNotInMs().add(pr); It is not a good practice to change internal states like this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19174) Enhance HMS to support drop of partitions with partition names as input
Janaki Lahorani created HIVE-19174: -- Summary: Enhance HMS to support drop of partitions with partition names as input Key: HIVE-19174 URL: https://issues.apache.org/jira/browse/HIVE-19174 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-19160) Insert data into decimal column fails with Null Pointer Exception
Janaki Lahorani created HIVE-19160: -- Summary: Insert data into decimal column fails with Null Pointer Exception Key: HIVE-19160 URL: https://issues.apache.org/jira/browse/HIVE-19160 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani drop table if exists testDecimal; create table testDecimal (cIdTINYINT, cBigIntDECIMAL, cInt DECIMAL, cSmallInt DECIMAL, cTinyint DECIMAL); insert into testDecimal values (1, 1234567890123456789, 1234567890, 12345, 123); insert into testDecimal values (2, 1, 2, 3, 4); The second insert fails with null pointer exception. 2018-04-10T15:23:23,080 ERROR [5dba40ef-be49-4187-8a72-afbb46c41ecc main] metastore.RetryingHMSHandler: java.lang.NullPointerException at org.apache.hadoop.hive.metastore.api.Decimal.compareTo(Decimal.java:318) at org.apache.hadoop.hive.metastore.columnstats.merge.DecimalColumnStatsMerger.merge(DecimalColumnStatsMerger.java:35) at org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.mergeColStats(MetaStoreUtils.java:1040) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:7166) at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108) at com.sun.proxy.$Proxy40.set_aggr_stats_for(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1870) at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.setPartitionColumnStatistics(SessionHiveMetaStoreClient.java:395) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212) at com.sun.proxy.$Proxy41.setPartitionColumnStatistics(Unknown Source) at org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4171) at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:179) at org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:83) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18718) Integer like types throws error when there is a match
Janaki Lahorani created HIVE-18718: -- Summary: Integer like types throws error when there is a match Key: HIVE-18718 URL: https://issues.apache.org/jira/browse/HIVE-18718 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani If a value is saved with long type and read as int type it results in FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18586) Upgrade Derby to 10.14.1.0
Janaki Lahorani created HIVE-18586: -- Summary: Upgrade Derby to 10.14.1.0 Key: HIVE-18586 URL: https://issues.apache.org/jira/browse/HIVE-18586 Project: Hive Issue Type: Improvement Reporter: Janaki Lahorani Assignee: Janaki Lahorani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18499) Amend point lookup tests to check for data
Janaki Lahorani created HIVE-18499: -- Summary: Amend point lookup tests to check for data Key: HIVE-18499 URL: https://issues.apache.org/jira/browse/HIVE-18499 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18472) Beeline gives log4j warnings
Janaki Lahorani created HIVE-18472: -- Summary: Beeline gives log4j warnings Key: HIVE-18472 URL: https://issues.apache.org/jira/browse/HIVE-18472 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Janaki Lahorani Starting Beeline gives the following warnings multiple times: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.x-1.cdh6.x.p0.215261/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See [http://www.slf4j.org/codes.html#multiple_bindings] for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18471) Beeline gives log4j warnings
Janaki Lahorani created HIVE-18471: -- Summary: Beeline gives log4j warnings Key: HIVE-18471 URL: https://issues.apache.org/jira/browse/HIVE-18471 Project: Hive Issue Type: Bug Affects Versions: 3.0.0 Reporter: Janaki Lahorani Starting Beeline gives the following warnings: -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-18393) Error returned when some other type is read as string from parquet tables
Janaki Lahorani created HIVE-18393: -- Summary: Error returned when some other type is read as string from parquet tables Key: HIVE-18393 URL: https://issues.apache.org/jira/browse/HIVE-18393 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani TimeStamp, Decimal, Double, Float, BigInt, Int, SmallInt, Tinyint and Boolean when read as String, Varchar or Char should return the correct data. Now this results in error for parquet tables. Test Case: drop table if exists testAltCol; create table testAltCol (cIdTINYINT, cTimeStamp TIMESTAMP, cDecimal DECIMAL(38,18), cDoubleDOUBLE, cFloat FLOAT, cBigIntBIGINT, cInt INT, cSmallInt SMALLINT, cTinyint TINYINT, cBoolean BOOLEAN); insert into testAltCol values (1, '2017-11-07 09:02:49.9', 12345678901234567890.123456789012345678, 1.79e308, 3.4e38, 1234567890123456789, 1234567890, 12345, 123, TRUE); insert into testAltCol values (2, '1400-01-01 01:01:01.1', 1.1, 2.2, 3.3, 1, 2, 3, 4, FALSE); insert into testAltCol values (3, '1400-01-01 01:01:01.1', 10.1, 20.2, 30.3, 1234567890123456789, 1234567890, 12345, 123, TRUE); select cId, cTimeStamp from testAltCol order by cId; select cId, cDecimal, cDouble, cFloat from testAltCol order by cId; select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltCol order by cId; select cId, cBoolean from testAltCol order by cId; drop table if exists testAltColP; create table testAltColP stored as parquet as select * from testAltCol; select cId, cTimeStamp from testAltColP order by cId; select cId, cDecimal, cDouble, cFloat from testAltColP order by cId; select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId; select cId, cBoolean from testAltColP order by cId; alter table testAltColP replace columns (cIdTINYINT, cTimeStamp STRING, cDecimal STRING, cDoubleSTRING, cFloat STRING, cBigIntSTRING, cInt STRING, cSmallInt STRING, cTinyint STRING, cBoolean STRING); select cId, cTimeStamp from testAltColP order by cId; select cId, cDecimal, cDouble, cFloat from testAltColP order by cId; select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId; select cId, cBoolean from testAltColP order by cId; alter table testAltColP replace columns (cIdTINYINT, cTimeStamp VARCHAR(100), cDecimal VARCHAR(100), cDoubleVARCHAR(100), cFloat VARCHAR(100), cBigIntVARCHAR(100), cInt VARCHAR(100), cSmallInt VARCHAR(100), cTinyint VARCHAR(100), cBoolean VARCHAR(100)); select cId, cTimeStamp from testAltColP order by cId; select cId, cDecimal, cDouble, cFloat from testAltColP order by cId; select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId; select cId, cBoolean from testAltColP order by cId; alter table testAltColP replace columns (cIdTINYINT, cTimeStamp CHAR(100), cDecimal CHAR(100), cDoubleCHAR(100), cFloat CHAR(100), cBigIntCHAR(100), cInt CHAR(100), cSmallInt CHAR(100), cTinyint CHAR(100), cBoolean CHAR(100)); select cId, cTimeStamp from testAltColP order by cId; select cId, cDecimal, cDouble, cFloat from testAltColP order by cId; select cId, cBigInt, cInt, cSmallInt, cTinyint from testAltColP order by cId; select cId, cBoolean from testAltColP order by cId; drop table if exists testAltColP; Error: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask Excerpt for log: 2018-01-05T15:54:05,756 ERROR [LocalJobRunner Map Task Executor #0] mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row [Error getting row data with exception java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.hive.serde2.io.TimestampWritable at org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetStringInspector.getPrimitiveJavaObject(ParquetStringInspector.java:77) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18311) Enable smb_mapjoin_8.q for cli driver
Janaki Lahorani created HIVE-18311: -- Summary: Enable smb_mapjoin_8.q for cli driver Key: HIVE-18311 URL: https://issues.apache.org/jira/browse/HIVE-18311 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18248) Clean up parameters
Janaki Lahorani created HIVE-18248: -- Summary: Clean up parameters Key: HIVE-18248 URL: https://issues.apache.org/jira/browse/HIVE-18248 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani Clean up of parameters that need not change at run time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18166) Result of hive.query.string is encoded.
Janaki Lahorani created HIVE-18166: -- Summary: Result of hive.query.string is encoded. Key: HIVE-18166 URL: https://issues.apache.org/jira/browse/HIVE-18166 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani set hive.query.string returns encoded string. hive.query.string=%0A%0Aselect+*+from+t1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-18147) Tests can fail with java.net.BindException: Address already in use
Janaki Lahorani created HIVE-18147: -- Summary: Tests can fail with java.net.BindException: Address already in use Key: HIVE-18147 URL: https://issues.apache.org/jira/browse/HIVE-18147 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Assignee: Janaki Lahorani Between the time a new port is picked up to start HMS or HS2, and the time when HMS or HS2 is started, the port can get used by some other service resulting in test failure. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17942) HiveAlterHandler not using conf from threadlocal
Janaki Lahorani created HIVE-17942: -- Summary: HiveAlterHandler not using conf from threadlocal Key: HIVE-17942 URL: https://issues.apache.org/jira/browse/HIVE-17942 Project: Hive Issue Type: Bug Affects Versions: 2.1.1 Reporter: Janaki Lahorani When HiveAlterHandler looks for conf, it is not getting the one from thread local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17832) Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session
Janaki Lahorani created HIVE-17832: -- Summary: Allow hive.metastore.disallow.incompatible.col.type.changes to be changed within a session Key: HIVE-17832 URL: https://issues.apache.org/jira/browse/HIVE-17832 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17764) alter view fails when hive.metastore.disallow.incompatible.col.type.changes set to true
Janaki Lahorani created HIVE-17764: -- Summary: alter view fails when hive.metastore.disallow.incompatible.col.type.changes set to true Key: HIVE-17764 URL: https://issues.apache.org/jira/browse/HIVE-17764 Project: Hive Issue Type: Bug Affects Versions: 2.1.1 Reporter: Janaki Lahorani Assignee: Janaki Lahorani A view is a virtual structure that derives the type information from the table(s) the view is based on.If the view definition is altered, the corresponding column types should be updated. The relevance of the change depending on the previous structure of the view is irrelevant. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17398) Support Costing/Heuristics to enable or disable DPP
Janaki Lahorani created HIVE-17398: -- Summary: Support Costing/Heuristics to enable or disable DPP Key: HIVE-17398 URL: https://issues.apache.org/jira/browse/HIVE-17398 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17396) Support DPP with map joins where the source and target belong in the same stage
Janaki Lahorani created HIVE-17396: -- Summary: Support DPP with map joins where the source and target belong in the same stage Key: HIVE-17396 URL: https://issues.apache.org/jira/browse/HIVE-17396 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17200) TestMiniSparkOnYarnCliDriver: [spark_vectorized_dynamic_partition_pruning]
Janaki Lahorani created HIVE-17200: -- Summary: TestMiniSparkOnYarnCliDriver: [spark_vectorized_dynamic_partition_pruning] Key: HIVE-17200 URL: https://issues.apache.org/jira/browse/HIVE-17200 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Error: Client execution failed with error code = 3 running " select count(*) from srcpart join (select ds as ds, ds as `date` from srcpart group by ds) s on (srcpart.ds = s.ds) where s.`date` = '2008-04-08'" fname=spark_vectorized_dynamic_partition_pruning.q -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17199) TestHiveMetaStoreStatsMerge: testStatsMerge
Janaki Lahorani created HIVE-17199: -- Summary: TestHiveMetaStoreStatsMerge: testStatsMerge Key: HIVE-17199 URL: https://issues.apache.org/jira/browse/HIVE-17199 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Stack: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_aggr_stats_for(ThriftHiveMetastore.java:3724) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_aggr_stats_for(ThriftHiveMetastore.java:3711) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1704) at org.apache.hadoop.hive.metastore.TestHiveMetaStoreStatsMerge.testStatsMerge(TestHiveMetaStoreStatsMerge.java:187) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17198) Flaky test: TestBeeLineDriver [smb_mapjoin_7]
Janaki Lahorani created HIVE-17198: -- Summary: Flaky test: TestBeeLineDriver [smb_mapjoin_7] Key: HIVE-17198 URL: https://issues.apache.org/jira/browse/HIVE-17198 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Error: Exception running or analyzing the results of the query file: org.apache.hive.beeline.QFile@4f7b68ad -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17197) Flaky test: TestMiniSparkOnYarnCliDriver
Janaki Lahorani created HIVE-17197: -- Summary: Flaky test: TestMiniSparkOnYarnCliDriver Key: HIVE-17197 URL: https://issues.apache.org/jira/browse/HIVE-17197 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Error: Failed during createSources processLine with code=3 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17057) Flaky test: TestHCatClient.testTableSchemaPropagation,testPartitionRegistrationWithCustomSchema,testPartitionSpecRegistrationWithCustomSchema
Janaki Lahorani created HIVE-17057: -- Summary: Flaky test: TestHCatClient.testTableSchemaPropagation,testPartitionRegistrationWithCustomSchema,testPartitionSpecRegistrationWithCustomSchema Key: HIVE-17057 URL: https://issues.apache.org/jira/browse/HIVE-17057 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17056) TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
Janaki Lahorani created HIVE-17056: -- Summary: TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic] Key: HIVE-17056 URL: https://issues.apache.org/jira/browse/HIVE-17056 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Client Execution succeeded but contained differences (error code = 1) after executing orc_ppd_basic.q 1287a1288 > Stage-1 LLAP IO COUNTERS: 1317a1319 > Stage-1 LLAP IO COUNTERS: 1338a1341 > Stage-1 LLAP IO COUNTERS: 1342a1346 > Stage-1 LLAP IO COUNTERS: -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17055) Flaky test: TestMiniLlapCliDriver.testCliDriver[llap_smb]
Janaki Lahorani created HIVE-17055: -- Summary: Flaky test: TestMiniLlapCliDriver.testCliDriver[llap_smb] Key: HIVE-17055 URL: https://issues.apache.org/jira/browse/HIVE-17055 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani Client Execution succeeded but contained differences (error code = 1) after executing llap_smb.q 324,325c324,325 < 2000 9 52 < 2001 0 139630 --- > 2001 4 139630 > 2001 6 52 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-17046) Flaky test: TestCliDriver[ppd_windowing2]
Janaki Lahorani created HIVE-17046: -- Summary: Flaky test: TestCliDriver[ppd_windowing2] Key: HIVE-17046 URL: https://issues.apache.org/jira/browse/HIVE-17046 Project: Hive Issue Type: Sub-task Reporter: Janaki Lahorani -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (HIVE-16960) Hive throws an ugly error exception when HDFS sticky bit is set
Janaki Lahorani created HIVE-16960: -- Summary: Hive throws an ugly error exception when HDFS sticky bit is set Key: HIVE-16960 URL: https://issues.apache.org/jira/browse/HIVE-16960 Project: Hive Issue Type: Bug Reporter: Janaki Lahorani Priority: Critical When calling LOAD DATA INPATH ... OVERWRITE INTO TABLE ... from a Hive user other than the HDFS file owner, and the HDFS sticky bit is set, then Hive will throw an error exception message that the file cannot be moved due to permission issues. Caused by: org.apache.hadoop.security.AccessControlException: Permission denied by sticky bit setting: user=hive, inode=sasdata-2016-04-20-17-13-43-630-e-1.dlv.bk The permission denied is expected, but the error message does not make sense to users + the stack trace displayed is huge. We should display a better error message to users, and maybe provide with help information about how to fix it. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
