[jira] [Created] (HIVE-13362) Commit binary file required for HIVE-13361
Prasanth Jayachandran created HIVE-13362: Summary: Commit binary file required for HIVE-13361 Key: HIVE-13362 URL: https://issues.apache.org/jira/browse/HIVE-13362 Project: Hive Issue Type: Sub-task Affects Versions: 2.0.0, 1.3.0, 2.1.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran Separate jira for committing orc binary file required for precommit test run for HIVE-13361 NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13361) Orc concatenation should enforce the compression buffer size
Prasanth Jayachandran created HIVE-13361:
Summary: Orc concatenation should enforce the compression buffer
size
Key: HIVE-13361
URL: https://issues.apache.org/jira/browse/HIVE-13361
Project: Hive
Issue Type: Bug
Affects Versions: 2.0.0, 1.3.0, 2.1.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Critical
With HIVE-11807 buffer size estimation happens by default. This can have
undesired effect wrt file concatenation. Consider the following table with files
{code}
testtable
-- 00_0 (created before HIVE-11807 which has buffer size 256KB)
-- 01_0 (created before HIVE-11807 which has buffer size 256KB)
-- 02_0 (created after HIVE-11807 with buffer size chosen as 128KB)
-- 03_0 (created after HIVE-11807 with buffer size chosen as 128KB)
{code}
If we perform ALTER TABLE .. CONCATENATE on the above table with HIVE-11807,
then depending on the split arrangement 00_0 and 01_0 will be
concatenated together to new merged file. But this new merged file will have
128KB buffer size (estimated buffer size and not requested buffer size). Since
new ORC writer size does not honor the requested buffer size the new merged
files will have smaller buffers than the required 256KB making the file
unreadable. Following exception will be thrown when reading the table after
concatenation
{code}
2016-03-24T16:26:33,974 ERROR [a9e27a9a-37cb-411d-9708-6c58a4ce34f2 main]:
CliDriver (SessionState.java:printError(1049)) - Failed with exception
java.io.IOException:java.lang.IllegalArgumentException: Buffer size too small.
size = 131072 needed = 153187
java.io.IOException: java.lang.IllegalArgumentException: Buffer size too small.
size = 131072 needed = 153187
at
org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:513)
at
org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:420)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:145)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1848)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:256)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:782)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13360) Refactoring Hive Authorization
Pengcheng Xiong created HIVE-13360: -- Summary: Refactoring Hive Authorization Key: HIVE-13360 URL: https://issues.apache.org/jira/browse/HIVE-13360 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13359) NoClassFoundError hadoop configuration with jdbc-standalone JAR
Rohit Dholakia created HIVE-13359: - Summary: NoClassFoundError hadoop configuration with jdbc-standalone JAR Key: HIVE-13359 URL: https://issues.apache.org/jira/browse/HIVE-13359 Project: Hive Issue Type: Bug Reporter: Rohit Dholakia Assignee: Rohit Dholakia When the hive-jdbc-SNAPSHOT-standalone.jar is used to run queries, it leads to a NoClassDefFoundError for org/apache/hadoop/conf/Configuration. This patch will resolve it by updating the jdbc/pom.xml file to not exclude commons-configuration and org.apache.hadoop:* as part of the maven shaded plugin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13358) stats state is not captured correctly: turn off stats optimizer for sampled table
Pengcheng Xiong created HIVE-13358: -- Summary: stats state is not captured correctly: turn off stats optimizer for sampled table Key: HIVE-13358 URL: https://issues.apache.org/jira/browse/HIVE-13358 Project: Hive Issue Type: Sub-task Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13357) TxnHandler.checkQFileTestHack() should not call TxnDbUtil.setConfValues()
Eugene Koifman created HIVE-13357: - Summary: TxnHandler.checkQFileTestHack() should not call TxnDbUtil.setConfValues() Key: HIVE-13357 URL: https://issues.apache.org/jira/browse/HIVE-13357 Project: Hive Issue Type: Improvement Affects Versions: 2.0.0, 1.3.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Fix For: 1.3.0, 2.1.0 this should be client side (test side) responsibility as is, it can sometimes clobber settings made by test client Longer term we should try not calling TxnDbUtil.prepDb(); from TxnHandler either. Can probably create a UDF to run this so that Q file tests can init the tables See if this is even necessary - all TXN tables are part main Derby .sql init file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13356) ClassCastException: org.apache.hadoop.hive.serde2.io.DateWritable cannot be cast to org.apache.hadoop.io.IntWritable
Vitalii Diravka created HIVE-13356:
--
Summary: ClassCastException:
org.apache.hadoop.hive.serde2.io.DateWritable cannot be cast to
org.apache.hadoop.io.IntWritable
Key: HIVE-13356
URL: https://issues.apache.org/jira/browse/HIVE-13356
Project: Hive
Issue Type: Bug
Affects Versions: 1.2.1
Reporter: Vitalii Diravka
Hive query involves cast date to int data type throughs cast exception.
{code}
[mapr@cs2none500 bin]$ rpm -qa |grep mapr
mapr-hadoop-core-2.7.0.32987.GA-1.x86_64
mapr-fileserver-5.0.0.32987.GA-1.x86_64
mapr-zookeeper-5.0.0.32987.GA-1.x86_64
mapr-historyserver-2.7.0.32987.GA-1.x86_64
mapr-hiveserver2-1.2.201601281422-1.noarch
mapr-hive-1.2.201601281422-1.noarch
mapr-hivemetastore-1.2.201601281422-1.noarch
mapr-core-internal-5.0.0.32987.GA-1.x86_64
mapr-mapreduce2-2.7.0.32987.GA-1.x86_64
mapr-core-5.0.0.32987.GA-1.x86_64
mapr-tasktracker-5.0.0.32987.GA-1.x86_64
mapr-jobtracker-5.0.0.32987.GA-1.x86_64
mapr-webserver-5.0.0.32987.GA-1.x86_64
mapr-nodemanager-2.7.0.32987.GA-1.x86_64
mapr-zk-internal-5.0.0.32987.GA.v3.4.5-1.x86_64
mapr-mapreduce1-0.20.2.32987.GA-1.x86_64
mapr-cldb-5.0.0.32987.GA-1.x86_64
mapr-resourcemanager-2.7.0.32987.GA-1.x86_64
mapr-drill-1.4.0.201601071151-1.noarch
{code}
Hive schema (parquet file in attachments):
{code}
create external table if not exists hive_storage.fewtypes_parquet (
int_col int,
bigint_col bigint,
date_col int,
time_col int,
timestamp_col bigint,
interval_col string,
varchar_col string,
float_col float,
double_col double,
bool_col boolean
)
STORED AS PARQUET
LOCATION '/tmp/testdata/hive_storage/fewtypes_parquet';
{code}
{code}
hive> use hive_storage;
OK
Time taken: 0.035 seconds
hive> show tables;
OK
fewtypes_parquet
Time taken: 0.034 seconds, Fetched: 1 row(s)
hive> select * from fewtypes_parquet;
OK
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
details.
Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: org.apache.hadoop.hive.serde2.io.DateWritable
cannot be cast to org.apache.hadoop.io.IntWritable
Time taken: 0.41 seconds
{code}
If date_col in CTAS is date the query will be executed.
Hive1.0 hasn't this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-13355) SparkCliDriver tests frequently timeout
Szehon Ho created HIVE-13355: Summary: SparkCliDriver tests frequently timeout Key: HIVE-13355 URL: https://issues.apache.org/jira/browse/HIVE-13355 Project: Hive Issue Type: Bug Reporter: Szehon Ho Something has changed recently to make the SparkCliDriver tests timeout frequently. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13354) Add ability to specify Compaction options per table
Eugene Koifman created HIVE-13354: - Summary: Add ability to specify Compaction options per table Key: HIVE-13354 URL: https://issues.apache.org/jira/browse/HIVE-13354 Project: Hive Issue Type: Improvement Affects Versions: 2.0.0, 1.3.0 Reporter: Eugene Koifman Assignee: Eugene Koifman Currently the are a few options that determine when automatic compaction is triggered. They are specified once for the warehouse. This doesn't make sense - some table may be more important and need to be compacted more often. We should allow specifying these on per table basis. Also, compaction is an MR job launched from within the metastore. There is currently no way to control job parameters (like memory, for example) except to specify it in hive-site.xml for metastore which means they are site wide. Should add a way to specify these per table (perhaps even per compaction if launched via ALTER TABLE) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13353) SHOW COMPACTIONS should support filtering options
Eugene Koifman created HIVE-13353: - Summary: SHOW COMPACTIONS should support filtering options Key: HIVE-13353 URL: https://issues.apache.org/jira/browse/HIVE-13353 Project: Hive Issue Type: Bug Components: Transactions Affects Versions: 2.0.0, 1.3.0 Reporter: Eugene Koifman Assignee: Eugene Koifman This command should support ability to filter by db/table/partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13352) Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others.
Aihua Xu created HIVE-13352: --- Summary: Seems unnecessary for HBase tests to call QTestUtil.tearDown to close zookeeper and others. Key: HIVE-13352 URL: https://issues.apache.org/jira/browse/HIVE-13352 Project: Hive Issue Type: Improvement Components: Test Affects Versions: 2.1.0 Reporter: Aihua Xu HBase tests right now call QTestUtil.tearDown to turn off Zookeeper and others after each test. Seems we can reuse them for all the tests while we just need to clear all the test data similar to TestCliDriver. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
Hari Sankar Sivarama Subramaniyan created HIVE-13351: Summary: Support drop Primary Key/Foreign Key constraints Key: HIVE-13351 URL: https://issues.apache.org/jira/browse/HIVE-13351 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13350) Support Alter commands for Rely/NoRely novalidate
Hari Sankar Sivarama Subramaniyan created HIVE-13350: Summary: Support Alter commands for Rely/NoRely novalidate Key: HIVE-13350 URL: https://issues.apache.org/jira/browse/HIVE-13350 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan Assignee: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-13349) Metastore Changes : HS2 changes : API calls for retrieving primary keys and foreign keys information
Hari Sankar Sivarama Subramaniyan created HIVE-13349: Summary: Metastore Changes : HS2 changes : API calls for retrieving primary keys and foreign keys information Key: HIVE-13349 URL: https://issues.apache.org/jira/browse/HIVE-13349 Project: Hive Issue Type: Sub-task Reporter: Hari Sankar Sivarama Subramaniyan -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Review Request 45274: HIVE-13290 : Support PK/FK constraint as part of create table command
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/45274/ --- Review request for hive and Ashutosh Chauhan. Repository: hive-git Description --- Support primary keys/foreign keys constraint as part of create table command in Hive Diffs - hcatalog/core/src/test/java/org/apache/hive/hcatalog/common/TestHCatUtil.java 102d6d2 itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java 6caf3fe itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggrStatsCacheIntegration.java 51d96dd itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseImport.java af60660 itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseSchemaTool.java e5833b8 itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestHBaseStoreIntegration.java 2cc1373 itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/hbase/TestStorageDescriptorSharing.java c29e46a metastore/if/hive_metastore.thrift 6a55962 metastore/scripts/upgrade/derby/034-HIVE-13076.derby.sql PRE-CREATION metastore/scripts/upgrade/derby/hive-schema-2.1.0.derby.sql 42f4eb6 metastore/scripts/upgrade/derby/upgrade-2.0.0-to-2.1.0.derby.sql a0bac3c metastore/scripts/upgrade/mssql/hive-schema-2.1.0.mssql.sql cf5a662 metastore/scripts/upgrade/mssql/upgrade-2.0.0-to-2.1.0.mssql.sql f25daf2 metastore/scripts/upgrade/mysql/034-HIVE-13076.mysql.sql PRE-CREATION metastore/scripts/upgrade/mysql/hive-schema-2.1.0.mysql.sql 6fd3209 metastore/scripts/upgrade/mysql/upgrade-2.0.0-to-2.1.0.mysql.sql e790636 metastore/scripts/upgrade/oracle/034-HIVE-13076.oracle.sql PRE-CREATION metastore/scripts/upgrade/oracle/hive-schema-2.1.0.oracle.sql 774f6be metastore/scripts/upgrade/oracle/upgrade-2.0.0-to-2.1.0.oracle.sql 8368d08 metastore/scripts/upgrade/postgres/033-HIVE-13076.postgres.sql PRE-CREATION metastore/scripts/upgrade/postgres/hive-schema-2.1.0.postgres.sql 7463a37 metastore/scripts/upgrade/postgres/upgrade-2.0.0-to-2.1.0.postgres.sql 6172407 metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 6e5de20 metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h 97c07a5 metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp 2695ffa metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java bb6e584 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java 083d340 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java 9004457 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AggrStats.java bfd1206 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ClearFileMetadataRequest.java 657bb7b metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ColumnStatistics.java ba059fe metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java 2769845 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/DropPartitionsResult.java adef415 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java de588e2 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FireEventRequest.java 6b08234 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Function.java 5f8ce0d metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetAllFunctionsResponse.java f88e279 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprRequest.java 0236b4a metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataByExprResult.java 89eb819 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataRequest.java 2408ad1 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetFileMetadataResult.java 8946635 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsInfoResponse.java 629c042 metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetOpenTxnsResponse.java 9f57a4a metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetPrincipalsInRoleResponse.java 02c652a metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/GetRoleGrantsForPrincipalResponse.java c013e0e metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HeartbeatTxnRangeResponse.java b00fb9c metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/HiveObjectRef.java d7dd8ec metast
