[jira] [Created] (HIVE-22923) Extract cumulative cost metadata from HiveRelMdDistinctRowCount metadata provider
Jesus Camacho Rodriguez created HIVE-22923: -- Summary: Extract cumulative cost metadata from HiveRelMdDistinctRowCount metadata provider Key: HIVE-22923 URL: https://issues.apache.org/jira/browse/HIVE-22923 Project: Hive Issue Type: Improvement Components: CBO Reporter: Jesus Camacho Rodriguez Assignee: Jesus Camacho Rodriguez It should not contained there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22922) LLAP: ShuffleHandler may not find shuffle data if pod restarts in k8s
Prasanth Jayachandran created HIVE-22922: Summary: LLAP: ShuffleHandler may not find shuffle data if pod restarts in k8s Key: HIVE-22922 URL: https://issues.apache.org/jira/browse/HIVE-22922 Project: Hive Issue Type: Bug Reporter: Nita Dembla Assignee: Prasanth Jayachandran Executor logs shows "Invalid map id: TTP/1.1 500 Internal Server Error". This happens when executor pod restarts with same hostname and port, but missing shuffle data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22921) materialized_view_partitioned_3.q relies on hive.optimize.sort.dynamic.partition property
Jesus Camacho Rodriguez created HIVE-22921:
--
Summary: materialized_view_partitioned_3.q relies on
hive.optimize.sort.dynamic.partition property
Key: HIVE-22921
URL: https://issues.apache.org/jira/browse/HIVE-22921
Project: Hive
Issue Type: Test
Reporter: Jesus Camacho Rodriguez
Assignee: Vineet Garg
{{hive.optimize.sort.dynamic.partition}} was deprecated in favor of
{{hive.optimize.sort.dynamic.partition.threshold}} in HIVE-20703.
{{materialized_view_partitioned_3.q}} specifically tests
SortedDynPartitionOptimizer for MVs. We need to update the q test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22920) Add row format OpenCSVSerde to the metastore column managed list
Ramesh Kumar Thangarajan created HIVE-22920: --- Summary: Add row format OpenCSVSerde to the metastore column managed list Key: HIVE-22920 URL: https://issues.apache.org/jira/browse/HIVE-22920 Project: Hive Issue Type: Bug Reporter: Ramesh Kumar Thangarajan Assignee: Ramesh Kumar Thangarajan Add row format OpenCSVSerde to the metastore column managed list -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72159: StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72159/ --- Review request for hive. Repository: hive-git Description --- Initial commit Diffs - ql/src/java/org/apache/hadoop/hive/ql/security/authorization/StorageBasedAuthorizationProvider.java 2a52e8354b Diff: https://reviews.apache.org/r/72159/diff/1/ Testing --- Thanks, Oleksiy Sayankin
[jira] [Created] (HIVE-22919) StorageBasedAuthorizationProvider does not allow create databases after changing hive.metastore.warehouse.dir
Oleksiy Sayankin created HIVE-22919:
---
Summary: StorageBasedAuthorizationProvider does not allow create
databases after changing hive.metastore.warehouse.dir
Key: HIVE-22919
URL: https://issues.apache.org/jira/browse/HIVE-22919
Project: Hive
Issue Type: Bug
Reporter: Oleksiy Sayankin
Assignee: Oleksiy Sayankin
*ENVIRONMENT:*
Hive-2.3
*STEPS TO REPRODUCE:*
1. Configure Storage Based Authorization:
{code:xml}
hive.security.authorization.enabled
true
hive.security.metastore.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.authorization.manager
org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider
hive.security.metastore.authenticator.manager
org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator
hive.metastore.pre.event.listeners
org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener
{code}
2. Create a few directories, change owners and permissions to it:
{code:java}hadoop fs -mkdir /tmp/m1
hadoop fs -mkdir /tmp/m2
hadoop fs -mkdir /tmp/m3
hadoop fs -chown testuser1:testuser1 /tmp/m[1,3]
hadoop fs -chmod 700 /tmp/m[1-3]{code}
3. Check permissions:
{code:java}[test@node2 ~]$ hadoop fs -ls /tmp|grep m[1-3]
drwx-- - testuser1 testuser1 0 2020-02-11 10:25 /tmp/m1
drwx-- - test test 0 2020-02-11 10:25 /tmp/m2
drwx-- - testuser1 testuser1 1 2020-02-11 10:36 /tmp/m3
[test@node2 ~]${code}
4. Loggin into Hive CLI using embedded Hive Metastore as *"testuser1"* user,
with *"hive.metastore.warehouse.dir"* set to *"/tmp/m1"*:
{code:java}sudo -u testuser1 hive --hiveconf hive.metastore.uris= --hiveconf
hive.metastore.warehouse.dir=/tmp/m1{code}
5. Perform the next steps:
{code:sql}-- 1. Check "hive.metastore.warehouse.dir" value:
SET hive.metastore.warehouse.dir;
-- 2. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
does not have an access:
SET hive.metastore.warehouse.dir=/tmp/m2;
-- 3. Try to create a database:
CREATE DATABASE m2;
-- 4. Set "hive.metastore.warehouse.dir" to the path, to which "testuser1" user
has an access:
SET hive.metastore.warehouse.dir=/tmp/m3;
-- 5. Try to create a database:
CREATE DATABASE m3;{code}
*ACTUAL RESULT:*
Query 5 fails with an exception below. It does not handle
"hive.metastore.warehouse.dir" proprty:
{code:java}hive> -- 5. Try to create a database:
hive> CREATE DATABASE m3;
FAILED: HiveException org.apache.hadoop.security.AccessControlException: User
testuser1(user id 5001) does not have access to hdfs:/tmp/m2/m3.db
hive>{code}
*EXPECTED RESULT:*
Query 5 creates a database;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22918) Investigate empty bucket file creation for ACID tables
Marta Kuczora created HIVE-22918: Summary: Investigate empty bucket file creation for ACID tables Key: HIVE-22918 URL: https://issues.apache.org/jira/browse/HIVE-22918 Project: Hive Issue Type: Task Affects Versions: 4.0.0 Reporter: Marta Kuczora Assignee: Marton Bod When creating an insert-only bucketed table with 5 buckets, and we insert only one row to this table, Hive creates empty files for the other 4 buckets. This logic is in the code for ACID tables as well, but when checking the table's final directory after the insert, I found that only 1 files got created. When debugged this issue, I found that the empty files are created in the staging directory outside the delta directory, therefore they won't get copied by the move task to the final directory. This behavior seems broken, but not sure if we really need the empty files in this case. This Jira is about investigating whether or not we need these empty files for ACID tables and if we do, fix the code to have them for ACID tables as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HIVE-22917) Configuration for Hive to recognise non-empty destination folders
Marta Kuczora created HIVE-22917:
Summary: Configuration for Hive to recognise non-empty destination
folders
Key: HIVE-22917
URL: https://issues.apache.org/jira/browse/HIVE-22917
Project: Hive
Issue Type: Task
Reporter: Marta Kuczora
Assignee: Marta Kuczora
Currently Hive overwrites the LOCATION folder even if it is non-empty in case
of INSERT or CTAS.
Investigate this behavior and if we can introduce a switch whereby any
ALTER/INSERT or CTAS or CREATE or DROP operation / transaction would be aborted
if the switch is ON and the LOCATION clause points at a non-empty folder.
{noformat}
>> create table test (json_data string)
STORED AS TEXTFILE
LOCATION 'hdfs://host-10-17-102-132.coe.>ra.com:8020/tmp/test'
TBLPROPERTIES ('serialization.null.format' = '');
>> insert into test values('test0');
>> insert into test values('test1');
>> insert into test values('test2');
>> select * from test;
INFO : Compiling
command(queryId=hive_20200207150101_601d6dbc-99cb-446d-86ac-6f8ce5304681):
select * from test
INFO : Executing
command(queryId=hive_20200207150101_601d6dbc-99cb-446d-86ac-6f8ce5304681):
select * from test
INFO : Completed executing
command(queryId=hive_20200207150101_601d6dbc-99cb-446d-86ac-6f8ce5304681); Time
taken: 0.001 seconds
INFO : OK
-+
test.json_data
-+
test0
test1
test2
-+
>> select * from test_id2;
INFO : Compiling
command(queryId=hive_20200207145656_e99d1a0d-ea4c-4636-ae3a-dd930df14644):
select * from test_id2
INFO : Executing
command(queryId=hive_20200207145656_e99d1a0d-ea4c-4636-ae3a-dd930df14644):
select * from test_id2
INFO : Completed executing
command(queryId=hive_20200207145656_e99d1a0d-ea4c-4636-ae3a-dd930df14644); Time
taken: 0.001 seconds
INFO : OK
--+
test_id2.id
--+
1
13
14
--+
>> create table test2 (json_data int)
STORED AS TEXTFILE
LOCATION 'hdfs://host-10-17-102-132.coe.>ra.com:8020/tmp/test'
as SELECT * from test_id;
INFO : Completed executing
command(queryId=hive_20200207150303_cbb57a17-1242-46dc-a98e-addf50f01c5b); Time
taken: 13.137 seconds
INFO : OK
No rows affected (13.226 seconds)
SELECT * from test;
INFO : Compiling
command(queryId=hive_20200207150404_d0aabd08-a15f-4e6c-99a3-e607b8a6cfd3):
SELECT * from test
INFO : Executing
command(queryId=hive_20200207150404_d0aabd08-a15f-4e6c-99a3-e607b8a6cfd3):
SELECT * from test
INFO : Completed executing
command(queryId=hive_20200207150404_d0aabd08-a15f-4e6c-99a3-e607b8a6cfd3); Time
taken: 0.001 seconds
INFO : OK
-+
test.json_data
-+
1
13
14
-+
3 rows selected (0.081 seconds)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HIVE-22916) Upgrade to Hadoop 3.2.2 once it's released
László Bodor created HIVE-22916: --- Summary: Upgrade to Hadoop 3.2.2 once it's released Key: HIVE-22916 URL: https://issues.apache.org/jira/browse/HIVE-22916 Project: Hive Issue Type: Improvement Reporter: László Bodor -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 72154: Break up DDLSemanticAnalyzer - extract the rest of the Alter Table analyzers
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/72154/ --- Review request for hive and Zoltan Haindrich. Bugs: HIVE-22907 https://issues.apache.org/jira/browse/HIVE-22907 Repository: hive-git Description --- DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is to refactor it in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: have a separate class for each analyzers have a package for each operation, containing an analyzer, a description, and an operation, so the amount of classes under a package is more manageable Step #15: extract the rest of the alter table analyzers from DDLSemanticAnalyzer, and move them under the new package. Remove DDLSemanticAnalyzer. Diffs - ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableAnalyzer.java 0acd5011cc ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractAlterTableOperation.java 323cdc42d3 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/AbstractBaseAlterTableAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/drop/DropTableOperation.java f4d1a35d72 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableRenameDesc.java 091c146940 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableRenameOperation.java 73ea400dcc ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableSetOwnerDesc.java 1fb11ce7c3 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableSetOwnerOperation.java 02b9b91dfc ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableSetPropertiesDesc.java 2d615a64cc ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableSetPropertiesOperation.java ff6b08b5d5 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableTouchDesc.java 207f5b443f ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableTouchOperation.java 8d8ac20350 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableUnsetPropertiesDesc.java bea9a365e9 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/AlterTableUnsetPropertiesOperation.java 31a27fc8d2 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/PreInsertTableDesc.java bc378809ad ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/PreInsertTableOperation.java 928ace9ed1 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/TruncateTableDesc.java 04fa1b9901 ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/TruncateTableOperation.java 3c62b0a1fa ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/columnstats/AlterTableUpdateColumnStatistictAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/owner/AlterTableSetOwnerAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/properties/AbstractAlterTablePropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/properties/AbstractAlterTableSetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/properties/AbstractAlterTableUnsetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/properties/AlterTableSetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/properties/AlterTableUnsetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/rename/AbstractAlterTableRenameAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/rename/AlterTableRenameAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/touch/AlterTableTouchAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/misc/truncate/TruncateTableAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/table/partition/drop/AlterTableDropPartitionOperation.java 9d186db7af ql/src/java/org/apache/hadoop/hive/ql/ddl/view/drop/DropViewOperation.java a029b6cca4 ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/drop/DropMaterializedViewOperation.java 93b25d4be3 ql/src/java/org/apache/hadoop/hive/ql/ddl/view/properties/AlterViewSetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/view/properties/AlterViewUnsetPropertiesAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/view/rename/AlterViewRenameAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java d7b6eeae55 ql/src/java/org/apache/hadoop/hive/ql/exec/repl/util/ReplUtils.java fc7f226d77 ql/src/java/org/apache/hadoop/hive/ql/parse/AcidExportSemanticAnalyzer.java cc3c5da8ce
