[jira] [Created] (HIVE-21893) Handle concurrent writes when ACID tables are getting bootst
Sankar Hariappan created HIVE-21893: --- Summary: Handle concurrent writes when ACID tables are getting bootst Key: HIVE-21893 URL: https://issues.apache.org/jira/browse/HIVE-21893 Project: Hive Issue Type: Bug Reporter: Sankar Hariappan -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21892) Trusted domain authentication should look at X-Forwarded-For header as well
Prasanth Jayachandran created HIVE-21892: Summary: Trusted domain authentication should look at X-Forwarded-For header as well Key: HIVE-21892 URL: https://issues.apache.org/jira/browse/HIVE-21892 Project: Hive Issue Type: Bug Affects Versions: 4.0.0 Reporter: Prasanth Jayachandran Assignee: Prasanth Jayachandran HIVE-21783 added trusted domain authentication. However, it looks only at request.getRemoteAddr() which works in most cases where there are no intermediate forward/reverse proxies. In trusted domain scenarios, if there intermediate proxies, the proxies typically append its own ip address "X-Forwarded-For" header. The X-Forwarded-For will look like clientIp -> proxyIp1 -> proxyIp2. The left most ip address in the X-Forwarded-For represents the real client ip address. For such scenarios, add a config to optionally look at X-Forwarded-For header when available to determine the real client ip. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 70875: HIVE-21884 Scheduled query support
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70875/ --- Review request for hive. Bugs: HIVE-21884 https://issues.apache.org/jira/browse/HIVE-21884 Repository: hive-git Description (updated) --- wip patch; with a lot of FIXMEs and some missing parts generated sources and q.out changes are excluded from the patch Diffs (updated) - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 089b88cdb0 data/conf/llap/hive-site.xml 44ca6c9daf itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java 54306234bf itests/src/test/resources/testconfiguration.properties 3d8bae8609 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java d2c2ccd5ea metastore/pom.xml 419a125d75 metastore/scripts/upgrade/hive/hive-schema-4.0.0.hive.sql cf3bfaa2b1 pom.xml 0fd8b8799b ql/if/queryplan.thrift de21914ba9 ql/pom.xml 7c4d26f512 ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 6ac695d502 ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintTask.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/hooks/ScheduledQueryCreationRegistryHook.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java d412dd72d1 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 3ca45fe1fe ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 46fa5635eb ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g 58fe0cd32e ql/src/java/org/apache/hadoop/hive/ql/parse/ScheduledQueryAnalyzer.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 51a6b2a918 ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java ba3330bf58 ql/src/java/org/apache/hadoop/hive/ql/schq/IScheduledQueryMaintenanceService.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/schq/MetastoreBasedScheduledQueryService.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionContext.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryExecutionService.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/schq/ScheduledQueryMaintWork.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/parse/TestParseDriver.java c5d099dcd8 ql/src/test/org/apache/hadoop/hive/ql/schq/TestScheduledQueryService.java PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/schq/TestScheduledQueryStatements.java PRE-CREATION ql/src/test/queries/clientpositive/sysdb_schq.q PRE-CREATION service/src/java/org/apache/hive/service/server/HiveServer2.java 00f6e90f6e standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java f947ee1918 standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java aa7e8dfcbd standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java c0d891961a standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 7d5d5b7895 standalone-metastore/metastore-server/pom.xml 2eeefa3cf0 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 7d948e448b standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java 61019c6a1a standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/RawStore.java 6a93e264f1 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java a5d0c046d4 standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MScheduledExecution.java PRE-CREATION standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MScheduledQuery.java PRE-CREATION standalone-metastore/metastore-server/src/main/resources/package.jdo 117a2f825e standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql 02ff4aed8f standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql cd03ba7ec6 standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java f202832e0b standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 1a7ce04630 standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClientPreCatalog.java 4878a47b2d standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestMetastoreScheduledQueries.java PRE-CREATION
Review Request 70876: Break up DDLTask - cleanup
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70876/ --- Review request for hive and Zoltan Haindrich. Bugs: HIVE-21891 https://issues.apache.org/jira/browse/HIVE-21891 Repository: hive-git Description --- DDLTask was a huge class, more than 5000 lines long. The related DDLWork was also a huge class, which had a field for each DDL operation it supported. The goal was to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: have a separate class for each operation have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable make all the requests (DDLDesc subclasses) immutable - most of them are now DDLTask should be agnostic to the actual operations right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...) In the interim time when there were two DDLTask and DDLWork classes in the code base the new ones in the new package were called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes were in use. Step #12: rename DDLTask2 and DDLWork2, now that they are alone. Remove the old DDLDesc. Instead of registering, now DDLTask finds the DDLOperations, and registers them itself. Diffs - contrib/src/test/results/clientnegative/serde_regex.q.out 65b8e314fc hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateDatabaseHook.java 1f05b892d9 hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/CreateTableHook.java 53ca98394d hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzer.java 8e5db2d8a9 hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java a237db87d2 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenarios.java 1f41d46782 itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosAcrossInstances.java 406bec6ae8 itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcDriver2.java 79e4818eb0 itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook.java 414feecd85 itests/util/src/main/java/org/apache/hadoop/hive/ql/metadata/DummySemanticAnalyzerHook1.java 554aaf3682 pom.xml fa7a12349b ql/pom.xml 7c4d26f512 ql/src/java/org/apache/hadoop/hive/ql/Driver.java 18438aa592 ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 7636019770 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLDesc.java 66e5cb0fed ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperation.java d556d55f97 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLOperationContext.java d5969cd123 ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLTask2.java ecc550103c ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLWork2.java 98f1fbf540 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/AlterDatabaseDesc.java 46eb092243 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/AlterDatabaseOperation.java a3bc7b000f ql/src/java/org/apache/hadoop/hive/ql/ddl/database/CreateDatabaseDesc.java 2ba8b186ed ql/src/java/org/apache/hadoop/hive/ql/ddl/database/CreateDatabaseOperation.java ed8da551ff ql/src/java/org/apache/hadoop/hive/ql/ddl/database/DescDatabaseDesc.java 948e24bd4f ql/src/java/org/apache/hadoop/hive/ql/ddl/database/DescDatabaseOperation.java 801ac62987 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/DropDatabaseDesc.java 933c3c652f ql/src/java/org/apache/hadoop/hive/ql/ddl/database/DropDatabaseOperation.java d9e50083f0 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/LockDatabaseDesc.java 5194c890ac ql/src/java/org/apache/hadoop/hive/ql/ddl/database/LockDatabaseOperation.java cf01a31785 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowCreateDatabaseDesc.java 848bfab6ad ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowCreateDatabaseOperation.java 100ac95f40 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowDatabasesDesc.java e8f4e442d4 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/ShowDatabasesOperation.java 476762feb2 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/SwitchDatabaseDesc.java 258b0366dc ql/src/java/org/apache/hadoop/hive/ql/ddl/database/SwitchDatabaseOperation.java 6e3d4faaf1 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/UnlockDatabaseDesc.java f105bfaa81 ql/src/java/org/apache/hadoop/hive/ql/ddl/database/UnlockDatabaseOperation.java 91ca0b00d9 ql/src/java/org/apache/hadoop/hive/ql/ddl/function/DescFunctionDesc.java b84c630149
[jira] [Created] (HIVE-21891) Break up DDLTask - cleanup
Miklos Gergely created HIVE-21891: - Summary: Break up DDLTask - cleanup Key: HIVE-21891 URL: https://issues.apache.org/jira/browse/HIVE-21891 Project: Hive Issue Type: Sub-task Components: Hive Affects Versions: 3.1.1 Reporter: Miklos Gergely Assignee: Miklos Gergely Fix For: 4.0.0 DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also a huge class, which has a field for each DDL operation it supports. The goal is to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: * have a separate class for each operation * have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable * make all the requests (DDLDesc subclasses) immutable * DDLTask should be agnostic to the actual operations * right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...) In the interim time when there are two DDLTask and DDLWork classes in the code base the new ones in the new package are called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes are in use. Step #11: extract the operations that left from the old DDLTask, and move them under the new packages. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21890) Fix alter_partition_change_col.q qtest inclusion in minillaplocal.query.files
David Lavati created HIVE-21890:
---
Summary: Fix alter_partition_change_col.q qtest inclusion in
minillaplocal.query.files
Key: HIVE-21890
URL: https://issues.apache.org/jira/browse/HIVE-21890
Project: Hive
Issue Type: Bug
Reporter: David Lavati
Assignee: David Lavati
HIVE-20833 introduced alter_partition_change_col.q under
minillaplocal.query.files, however it was named only
\{{alter_partition_change_col}} without the postfix, thus the test never ran.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HIVE-21889) Add reexecution configuration keys to the whitelist
Zoltan Haindrich created HIVE-21889: --- Summary: Add reexecution configuration keys to the whitelist Key: HIVE-21889 URL: https://issues.apache.org/jira/browse/HIVE-21889 Project: Hive Issue Type: Improvement Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21888) Set hive.parquet.timestamp.skip.conversion default to true
Karen Coppage created HIVE-21888: Summary: Set hive.parquet.timestamp.skip.conversion default to true Key: HIVE-21888 URL: https://issues.apache.org/jira/browse/HIVE-21888 Project: Hive Issue Type: Bug Reporter: Karen Coppage Assignee: Karen Coppage -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: To contribute
Hi Nandini, You might want to ask this questions directly on the jira - the ones interested in most often are subscribed to them, so they will see your question. You can definitely assign this to yourself. As a first task I would check if this is still an existing issue, or not. Thanks, Peter > On Jun 18, 2019, at 13:57, Nandini Mankale wrote: > > Hello, > > I would like to check this task and work on it. > https://issues.apache.org/jira/browse/HIVE-18591 > > Can I assign it to myself? > > Thanks > Nandini > > > Sent from Mail for Windows 10 >
To contribute
Hello, I would like to check this task and work on it. https://issues.apache.org/jira/browse/HIVE-18591 Can I assign it to myself? Thanks Nandini Sent from Mail for Windows 10
[jira] [Created] (HIVE-21887) Multiple implementations of PersistenceManager are on the classpath
Zoltan Haindrich created HIVE-21887: --- Summary: Multiple implementations of PersistenceManager are on the classpath Key: HIVE-21887 URL: https://issues.apache.org/jira/browse/HIVE-21887 Project: Hive Issue Type: Bug Reporter: Zoltan Haindrich Assignee: Zoltan Haindrich While writing some test I've just bumped into that PersistenceManager is not always AutoCloseable ; however when I was using it from the metastore it was. it turned out that: * 'standalone-metastore' uses: org.datanucleus:javax.jdo:3.2.0-m3 * 'metastore' uses: javax.jdo:jdo-api:3.0.1 the problem is that both dependency contains the jdo api; and sometimes jdo-api is earlier on the classpath -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HIVE-21886) REPL - With table list - Handle rename events during replace policy
mahesh kumar behera created HIVE-21886: -- Summary: REPL - With table list - Handle rename events during replace policy Key: HIVE-21886 URL: https://issues.apache.org/jira/browse/HIVE-21886 Project: Hive Issue Type: Sub-task Components: repl Reporter: mahesh kumar behera Assignee: mahesh kumar behera REPL DUMP fetches the events from NOTIFICATION_LOG table based on regular expression + inclusion/exclusion list. So, in case of rename table event, the event will be ignored if old table doesn't match the pattern but the new table should be bootstrapped. REPL DUMP should have a mechanism to detect such tables and automatically bootstrap with incremental replication.Also, if renamed table is excluded from replication policy, then need to drop the old table at target as well. There are 4 scenarios that needs to be handled. # Both new name and old name satisfies the table name pattern filter. ## No need to do anything. The incremental event for rename should take care of the replication. # Both the names does not satisfy the table name pattern filter. ## Both the names are not in the scope of the policy and this nothing needs to be done. # New name satisfies the pattern but the old name does not. ## The table will not be present at the target. ## Rename event handler for dump should detect this case and add the new table name to the list of table for bootstrap. ## All the events related to the table (new name) should be ignored. ## If there is a drop event for the table (with new name), then remove the table from the list of tables to be bootstrapped. ## In case of rename (double rename) ### If the new name satisfies the table pattern, then add the new name to the list of tables to be bootstrapped and remove the old name from the list of tables to be bootstrapped. ### If the new name does not satisfies then just removed the table name from the list of tables to be bootstrapped. # New name does not satisfies the pattern but the old name satisfies. ## Change the rename event to a drop event. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Review Request 70873: Break up DDLTask - extract rest of the operations
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/70873/ --- Review request for hive and Zoltan Haindrich. Bugs: HIVE-21881 https://issues.apache.org/jira/browse/HIVE-21881 Repository: hive-git Description --- DDLTask is a huge class, more than 5000 lines long. The related DDLWork is also a huge class, which has a field for each DDL operation it supports. The goal is to refactor these in order to have everything cut into more handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: have a separate class for each operation have a package for each operation group (database ddl, table ddl, etc), so the amount of classes under a package is more manageable make all the requests (DDLDesc subclasses) immutable DDLTask should be agnostic to the actual operations right now let's ignore the issue of having some operations handled by DDLTask which are not actual DDL operations (lock, unlock, desc...) In the interim time when there are two DDLTask and DDLWork classes in the code base the new ones in the new package are called DDLTask2 and DDLWork2 thus avoiding the usage of fully qualified class names where both the old and the new classes are in use. Step #11: extract the operations that left from the old DDLTask, and move them under the new packages. Diffs - hcatalog/core/src/main/java/org/apache/hive/hcatalog/cli/SemanticAnalysis/HCatSemanticAnalyzerBase.java e84dfdc931 ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/CacheMetadataDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/CacheMetadataOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/InsertCommitHookOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/MsckDesc.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/MsckOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/ReplRemoveFirstIncLoadPendFlagOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/ShowConfOperation.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/package-info.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/ddl/view/CreateViewDesc.java 595da9c5ea ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 89dca2e7bd ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 04d12075a5 ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 6ac695d502 ql/src/java/org/apache/hadoop/hive/ql/exec/repl/incremental/IncrementalLoadTasksBuilder.java f6975d3362 ql/src/java/org/apache/hadoop/hive/ql/optimizer/QueryPlanPostProcessor.java 3f5b0e3e36 ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java f13593e36c ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1a2a1e8f47 ql/src/java/org/apache/hadoop/hive/ql/plan/CacheMetadataDesc.java e061adb757 ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 639aaba9e8 ql/src/java/org/apache/hadoop/hive/ql/plan/InsertCommitHookDesc.java 8136506381 ql/src/java/org/apache/hadoop/hive/ql/plan/MsckDesc.java 5d2307c168 ql/src/java/org/apache/hadoop/hive/ql/plan/ReplRemoveFirstIncLoadPendFlagDesc.java afa0a09af2 ql/src/java/org/apache/hadoop/hive/ql/plan/ShowConfDesc.java 9efecb1845 ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java 6ee74ca257 ql/src/test/queries/clientpositive/msck_repair_drop.q 407249e55a ql/src/test/queries/clientpositive/show_conf.q 064358d49d ql/src/test/queries/clientpositive/stats_filemetadata.q 5ddbaac626 ql/src/test/results/clientnegative/msck_repair_1.q.out 174419fe44 ql/src/test/results/clientnegative/msck_repair_2.q.out 174419fe44 ql/src/test/results/clientnegative/msck_repair_3.q.out 174419fe44 ql/src/test/results/clientnegative/msck_repair_4.q.out bb9cf47b08 ql/src/test/results/clientnegative/table_nonprintable_negative.q.out 3891747c8d ql/src/test/results/clientpositive/druid/druidmini_dynamic_partition.q.out 1401dbc375 ql/src/test/results/clientpositive/druid/druidmini_mv.q.out 08a83da67a ql/src/test/results/clientpositive/msck_repair_drop.q.out 27b718c670 ql/src/test/results/clientpositive/show_conf.q.out 7e65dfd31e Diff: https://reviews.apache.org/r/70873/diff/1/ Testing --- All the unit tests are still running + added some new init tests. Thanks, Miklos Gergely
Export control
In registering ORC as using encryption, I realised that hive isn't listed, although it has used encryption since 1.3. Since I just ran through the process, unless someone objects, I'll do it for hive also. .. Owen
