[jira] [Commented] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255160#comment-15255160 ] Pengcheng Xiong commented on HIVE-13565: Addressed [~ashutoshc]'s comments. Could u take another look? Thanks. > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Attachment: HIVE-13565.02.patch > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Open (was: Patch Available) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13565) thrift change
[ https://issues.apache.org/jira/browse/HIVE-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13565: --- Status: Patch Available (was: Open) > thrift change > - > > Key: HIVE-13565 > URL: https://issues.apache.org/jira/browse/HIVE-13565 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13565.01.patch, HIVE-13565.02.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13541) Pass view's ColumnAccessInfo to HiveAuthorizer
[
https://issues.apache.org/jira/browse/HIVE-13541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255147#comment-15255147
]
Hive QA commented on HIVE-13541:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800248/HIVE-13541.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 41 failed/errored test(s), 9943 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-cte_4.q-schema_evol_text_nonvec_mapwork_table.q-vector_groupby_reduce.q-and-12-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_decimal_2.q-explainuser_1.q-explainuser_3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testForcedLocalityPreemption
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.concurrencyFalse
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLExclusive
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLNoLock
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDDLShared
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testDelete
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testRollback
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadPartition
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleReadTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testSingleWriteTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testUpdate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testWriteDynamicPartition
org.apache.hadoop.hive.ql.parse.TestColumnAccess.testJoinView1AndTable2
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/45/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/45/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-45/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 41 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800248 -
[jira] [Updated] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner updated HIVE-13467: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master. > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Fix For: 2.1.0 > > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > HIVE-13467.6.patch, screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13559) Pass exception to failure hooks
[
https://issues.apache.org/jira/browse/HIVE-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255048#comment-15255048
]
Hive QA commented on HIVE-13559:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12799798/HIVE-13559.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 58 failed/errored test(s), 9948 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket5
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_bucket6
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_disable_merge_for_bucketing
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_map_operators
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_num_buckets
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_infer_bucket_sort_reducers_power_two
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge9
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_reduce_deduplicate
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join2
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join3
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_vector_outer_join5
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreAuthorization.testMetaStoreAuthorization
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hadoop.hive.ql.TestTxnCommands2.testOrcNoPPD
org.apache.hadoop.hive.ql.TestTxnCommands2.testUpdateMixedCase
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
[jira] [Updated] (HIVE-13445) LLAP: token should encode application and cluster ids
[ https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13445: Attachment: HIVE-13445.03.patch Some changes were reverted in error > LLAP: token should encode application and cluster ids > - > > Key: HIVE-13445 > URL: https://issues.apache.org/jira/browse/HIVE-13445 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch, > HIVE-13445.03.patch, HIVE-13445.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.02.patch > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.02.patch, > HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255032#comment-15255032
]
Sergey Shelukhin commented on HIVE-13447:
-
No, there's a separate (linked) JIRA for that
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13446) LLAP: set default management protocol acls to deny all
[ https://issues.apache.org/jira/browse/HIVE-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255020#comment-15255020 ] Siddharth Seth commented on HIVE-13446: --- The patch itself looks good in what it's doing. Couple of questions / comments. - Is the LLAP_VALIDATE_ACLS property really needed ? Why not always have this enabled. - Changing the default for "hive.llap.management.acl" to " " instead of "*" seems to be a simpler approach. Afaik, the logged in user will still be allowed access. The default would allow only the logged in user (assuming that works). Instead of changing LLAP_VALIDATE_ACLS - users can modify the actual ACLs if they want to grant access to additional users. - hive.llap.management.acl.blocked - This seems very brittle. BLOCKED is an internal constant in Hadoop ServiceAuthorizationManager. I'm not sure how any project outside of Hadoop is supposed to use this in a reliable manner. Maybe define the man acl configuration as a string and add the blocked to it - to prevent strange naming problems mentioned in the code. > LLAP: set default management protocol acls to deny all > -- > > Key: HIVE-13446 > URL: https://issues.apache.org/jira/browse/HIVE-13446 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13446.patch > > > The user needs to set the acls. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255018#comment-15255018
]
Siddharth Seth commented on HIVE-13447:
---
Does this also cover the ZK path used by the ZKSecretManager ?
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15255009#comment-15255009 ] Sergey Shelukhin commented on HIVE-13597: - [~thejas] can you take a look? > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HIVE-12892) Add global change versioning to permanent functions in metastore
[ https://issues.apache.org/jira/browse/HIVE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-12892: - > Add global change versioning to permanent functions in metastore > > > Key: HIVE-12892 > URL: https://issues.apache.org/jira/browse/HIVE-12892 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12892.01.patch, HIVE-12892.02.patch, > HIVE-12892.03.patch, HIVE-12892.04.patch, HIVE-12892.05.nogen.patch, > HIVE-12892.05.patch, HIVE-12892.05.patch, HIVE-12892.06.patch, > HIVE-12892.nogen.patch, HIVE-12892.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12892) Add global change versioning to permanent functions in metastore
[ https://issues.apache.org/jira/browse/HIVE-12892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-12892: Attachment: HIVE-12892.06.patch > Add global change versioning to permanent functions in metastore > > > Key: HIVE-12892 > URL: https://issues.apache.org/jira/browse/HIVE-12892 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-12892.01.patch, HIVE-12892.02.patch, > HIVE-12892.03.patch, HIVE-12892.04.patch, HIVE-12892.05.nogen.patch, > HIVE-12892.05.patch, HIVE-12892.05.patch, HIVE-12892.06.patch, > HIVE-12892.nogen.patch, HIVE-12892.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Attachment: HIVE-13597.patch A new patch; the revert causes a lot of conflicts, so the tests and review are probably needed. > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Status: Patch Available (was: Open) > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.nogen.patch, HIVE-13597.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Attachment: HIVE-13597.nogen.patch > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13597.nogen.patch > > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13597) revert HIVE-12892
[ https://issues.apache.org/jira/browse/HIVE-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13597: Target Version/s: 2.1.0 > revert HIVE-12892 > - > > Key: HIVE-13597 > URL: https://issues.apache.org/jira/browse/HIVE-13597 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > Updating functions in LLAP via a metastore call is not going to be committed > in the current form due to other design considerations. > For now, we should revert the changes made in HIVE-12892; there's a DB change > involved there, and we don't want that to be included in a released version > before we decide on the final design of the update feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13463: Attachment: HIVE-13463.4.patch Again > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463-4.patch, HIVE-13463.4.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13130: Status: Open (was: Patch Available) > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, HIVE-13130.5.patch > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13130: Status: Patch Available (was: Open) > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, HIVE-13130.5.patch > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13130) HS2 changes : API calls for retrieving primary keys and foreign keys information
[ https://issues.apache.org/jira/browse/HIVE-13130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13130: Attachment: HIVE-13130.5.patch > HS2 changes : API calls for retrieving primary keys and foreign keys > information > - > > Key: HIVE-13130 > URL: https://issues.apache.org/jira/browse/HIVE-13130 > Project: Hive > Issue Type: Sub-task >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13130.1.patch, HIVE-13130.2.patch, > HIVE-13130.3.patch, HIVE-13130.4.patch, HIVE-13130.5.patch, HIVE-13130.5.patch > > > ODBC exposes the SQLPrimaryKeys and SQLForeignKeys API calls and JDBC exposes > getPrimaryKeys and getCrossReference API calls. We need to provide these > interfaces as part of PK/FK implementation in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13407) Add more subtlety to TezCompiler Perf Logging
[ https://issues.apache.org/jira/browse/HIVE-13407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-13407: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Hari! > Add more subtlety to TezCompiler Perf Logging > - > > Key: HIVE-13407 > URL: https://issues.apache.org/jira/browse/HIVE-13407 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Fix For: 2.1.0 > > Attachments: HIVE-13407.1.patch, HIVE-13407.2.patch, > HIVE-13407.3.patch > > > We can add more subtlety to perf logging information in TezCompiler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13445) LLAP: token should encode application and cluster ids
[
https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254975#comment-15254975
]
Sergey Shelukhin edited comment on HIVE-13445 at 4/23/16 12:48 AM:
---
{noformat}
Any possibility of performing some basic sanity checks inside
LlapProtocolServerImpl - or is that already in place via the RPC layer
validating the presence of a LLAP token. Don't like the fact that the security
chceks are 3 calls deep - but that seems the best place for them rightnow.
{noformat}
The RPC layer validates the presence of the token.
{noformat}
String hostName = MetricsUtils.getHostName(); - Not necessarily related to this
patch, but getting it from YARN is more consistent (when yarn is available).
Have seen lots of issues around figuring out hostnames otherwise.
{noformat}
Is the yarn option already used somewhere? We could just change the utility
method to use it too.
{noformat}
LlapDeamon: appName = UUID.randomUUID().toString();
Ths won't work on distributed clusters, right ? Tokens use this as the
appSecret. Each node will generate a different appSecret. daemonId.getAppSecret
is being used as the clusterId in LlapTokenIdentifier.
{noformat}
We assume this is only used in tests. It won't work indeed. Added a comment
{noformat}
In LlapTokenChecker - why are we iterating over tokens even after an LLAPToken
has been found ? Are multiple tokens expected. This is in checkPermissions as
well as getTokenInfo
{noformat}
Not really expected at this point; I wonder if external clients could be using
something like that.
{noformat}
It looks like we end up taking the first request and linking it with the query.
Also subsequent requests are validated against this. Assuming that this becomes
more useful once signing comes in - to make sure someone is not submitting with
incorrect parameters.
{noformat}
Yes, if we also validate it against the signature. In general, though, we
assume that whoever can submit fragments (ie has the specific token) can also
kill fragments. The key is not being able to submit/kill/etc. fragments for an
app with a different token.
{noformat}
TaskExecutorService.findQueryByFragment - think we're better off implementing
this in QueryInfo itself rather than going to the scheduler to find out this
information. need to check if QueryInfo has state information about which
fragments are linked to a query.
{noformat}
It doesn't, as far as I can tell.
{noformat}
getDelegationToken(String appSecret) - even in case of Tez, should this be
associated with the sessionId. That prevents a lot of the if (token.appSecret
== null) checks and will simplify the code.
{noformat}
Don't understand. Can you elaborate?
{noformat}
Forgot to mention, we should add some tests to validate token functionality,
and how the system interacts with QueryInfo etc.
{noformat}
Separate JIRA?
{noformat}
More on this. If eventually, we're going to validate this via signatures for
external access - do we actually need to store the appSecret/appId for the
Query. Instead, we could validate future requests against the already stored
applicationId for a fragment / query.
{noformat}
The app ID has to come from somewhere with each request; terminate/etc.
requests themselves are not signed, so we cannot rely on the user to give us
the correct app Id to verify against the fragment. The appId when submitting
can indeed come from the signed message, not from the token (or it could be
verified to be the same from both).
But, I think we'd still need it in the token for other requests. I am actually
not sure how the token will work with signing right now, more specifically -
will we be able to get away with not having appsecret be a secret? I think we
will if HS2 would generate and sign it. However, if the client is allowed to
pass it in, some other client might also pass in the same appId (if it's not a
secret), and get the same token. So I assume we'd still store it, although it
won't really be called secret, it's just something that the signer (HS2) has to
generate.
Fixing the rest.
was (Author: sershe):
{noformat}
Any possibility of performing some basic sanity checks inside
LlapProtocolServerImpl - or is that already in place via the RPC layer
validating the presence of a LLAP token. Don't like the fact that the security
chceks are 3 calls deep - but that seems the best place for them rightnow.
{noformat}
The RPC layer validates the presence of the token.
{noformat}
String hostName = MetricsUtils.getHostName(); - Not necessarily related to this
patch, but getting it from YARN is more consistent (when yarn is available).
Have seen lots of issues around figuring out hostnames otherwise.
{noformat}
Is the yarn option already used somewhere? We could just change the utility
method to use it too.
{noformat}
LlapDeamon: appName = UUID.randomUUID().toString();
Ths won't work
[jira] [Updated] (HIVE-13445) LLAP: token should encode application and cluster ids
[ https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13445: Attachment: HIVE-13445.02.patch > LLAP: token should encode application and cluster ids > - > > Key: HIVE-13445 > URL: https://issues.apache.org/jira/browse/HIVE-13445 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch, > HIVE-13445.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13445) LLAP: token should encode application and cluster ids
[
https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254975#comment-15254975
]
Sergey Shelukhin commented on HIVE-13445:
-
{noformat}
Any possibility of performing some basic sanity checks inside
LlapProtocolServerImpl - or is that already in place via the RPC layer
validating the presence of a LLAP token. Don't like the fact that the security
chceks are 3 calls deep - but that seems the best place for them rightnow.
{noformat}
The RPC layer validates the presence of the token.
{noformat}
String hostName = MetricsUtils.getHostName(); - Not necessarily related to this
patch, but getting it from YARN is more consistent (when yarn is available).
Have seen lots of issues around figuring out hostnames otherwise.
{noformat}
Is the yarn option already used somewhere? We could just change the utility
method to use it too.
{noformat}
LlapDeamon: appName = UUID.randomUUID().toString();
Ths won't work on distributed clusters, right ? Tokens use this as the
appSecret. Each node will generate a different appSecret. daemonId.getAppSecret
is being used as the clusterId in LlapTokenIdentifier.
{noformat}
We assume this is only used in tests. It won't work indeed. Added a comment
{noformat}
In LlapTokenChecker - why are we iterating over tokens even after an LLAPToken
has been found ? Are multiple tokens expected. This is in checkPermissions as
well as getTokenInfo
{noformat}
Not really expected at this point; I wonder if external clients could be using
something like that.
{noformat}
It looks like we end up taking the first request and linking it with the query.
Also subsequent requests are validated against this. Assuming that this becomes
more useful once signing comes in - to make sure someone is not submitting with
incorrect parameters.
{noformat}
Yes, if we also validate it against the signature. In general, though, we
assume that whoever can submit fragments (ie has the specific token) can also
kill fragments. The key is not being able to submit/kill/etc. fragments for an
app with a different token.
{noformat}
TaskExecutorService.findQueryByFragment - think we're better off implementing
this in QueryInfo itself rather than going to the scheduler to find out this
information. need to check if QueryInfo has state information about which
fragments are linked to a query.
{noformat}
It doesn't, as far as I can tell.
{noformat}
getDelegationToken(String appSecret) - even in case of Tez, should this be
associated with the sessionId. That prevents a lot of the if (token.appSecret
== null) checks and will simplify the code.
{noformat}
Don't understand. Can you elaborate?
{noformat}
Forgot to mention, we should add some tests to validate token functionality,
and how the system interacts with QueryInfo etc.
{noformat}
Separate JIRA?
{noformat}
More on this. If eventually, we're going to validate this via signatures for
external access - do we actually need to store the appSecret/appId for the
Query. Instead, we could validate future requests against the already stored
applicationId for a fragment / query.
{noformat}
The app ID has to come from somewhere with each request; terminate/etc.
requests themselves are not signed. I am actually not sure how the token will
work with signing right now, more specifically - will we be able to get away
with not having appsecret be a secret? I think we will if HS2 would generate
and sign it. However, if the client is allowed to pass it in, some other client
might also pass in the same appId and secret, and get the same token. So I
assume we'd still store it, although it won't really be called secret, it's
just something that the signer (HS2) has to generate.
Fixing the rest.
> LLAP: token should encode application and cluster ids
> -
>
> Key: HIVE-13445
> URL: https://issues.apache.org/jira/browse/HIVE-13445
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13445.01.patch, HIVE-13445.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254968#comment-15254968 ] Gunther Hagleitner commented on HIVE-13467: --- Test failures are unrelated. > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > HIVE-13467.6.patch, screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254964#comment-15254964 ] Gunther Hagleitner commented on HIVE-13467: --- [~sseth] llap.html is static right now. It loads a JS file that will make a call to /llap servlet to fetch the data. That makes it difficult to let you switch cluster view by changing the url folks enter. I also think that if we need to support multiple clusters, we should have a proper way to expose this on the UI. We can have a "Change cluster" input field on the llap page. That way it's more straight forward for people to know what they're looking at. However, all that seems optional to me. The way to use llap right now is to set the instance you're using in the config of your hs2. That's the one we'll display by default. If you have multiple clusters, you'd have multiple hs2 instances likely anyways, no? > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > HIVE-13467.6.patch, screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13596) HS2 should refresh UDFs for each session, at least in multi-HS2 case
[ https://issues.apache.org/jira/browse/HIVE-13596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13596: Description: When multiple HS2s are run, creating a permanent fn is only executed on one of them, and the other HS2s don't get the new function. Unlike say with tables, where we always get stuff from db on demand, fns are registered at certain points in the code and if the new one is not registered, it will not be available. We could change the code to refresh the udf by name if it's missing, similar to getting a table or whatever; or we could refresh UDFs when a session is started in multi-HS2 case, or at some other convenient point. was: When multiple HS2s are run, creating a permanent fn is only executed on one of them, and the other HS2s don't get the new function. Unlike say with tables, where we always get stuff from db on demand, fns are registered at certain points in the code and if the new one is not registered, it will not be available. We could change the code to refresh the udf by name if it's missing, similar to getting a table or whatever; or we could refresh UDFs when a session is started in multi-HS2 case, or in other such logical point. > HS2 should refresh UDFs for each session, at least in multi-HS2 case > > > Key: HIVE-13596 > URL: https://issues.apache.org/jira/browse/HIVE-13596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > When multiple HS2s are run, creating a permanent fn is only executed on one > of them, and the other HS2s don't get the new function. Unlike say with > tables, where we always get stuff from db on demand, fns are registered at > certain points in the code and if the new one is not registered, it will not > be available. > We could change the code to refresh the udf by name if it's missing, similar > to getting a table or whatever; or we could refresh UDFs when a session is > started in multi-HS2 case, or at some other convenient point. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254954#comment-15254954 ] Aihua Xu commented on HIVE-13592: - You are right. That's a very nice way to handle the synchronization. +1. > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12159) Create vectorized readers for the complex types
[
https://issues.apache.org/jira/browse/HIVE-12159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254951#comment-15254951
]
Hive QA commented on HIVE-12159:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800243/HIVE-12159.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 9939 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_non_string_partition.q-delete_where_non_partitioned.q-auto_sortmerge_join_16.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
org.apache.hive.spark.client.TestSparkClient.testSyncRpc
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/43/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/43/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-43/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800243 -
[jira] [Commented] (HIVE-13346) LLAP doesn't update metadata priority when reusing from cache; some tweaks in LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254948#comment-15254948 ] Prasanth Jayachandran commented on HIVE-13346: -- +1 > LLAP doesn't update metadata priority when reusing from cache; some tweaks in > LRFU policy > - > > Key: HIVE-13346 > URL: https://issues.apache.org/jira/browse/HIVE-13346 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13346.01.patch, HIVE-13346.02.patch, > HIVE-13346.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13560) Adding Omid as connection manager for HBase Metastore
[ https://issues.apache.org/jira/browse/HIVE-13560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13560: -- Attachment: HIVE-13560.2.patch Reattach the same patch for precommit test. > Adding Omid as connection manager for HBase Metastore > - > > Key: HIVE-13560 > URL: https://issues.apache.org/jira/browse/HIVE-13560 > Project: Hive > Issue Type: Improvement > Components: HBase Metastore >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13560.1.patch, HIVE-13560.2.patch > > > Adding Omid as a transaction manager to HBase Metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13551) Make cleardanglingscratchdir work on Windows
[ https://issues.apache.org/jira/browse/HIVE-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-13551: -- Attachment: HIVE-13551.2.patch Attach same patch for precommit test. > Make cleardanglingscratchdir work on Windows > > > Key: HIVE-13551 > URL: https://issues.apache.org/jira/browse/HIVE-13551 > Project: Hive > Issue Type: Bug >Reporter: Daniel Dai >Assignee: Daniel Dai > Attachments: HIVE-13551.1.patch, HIVE-13551.2.patch > > > See a couple of issues when running cleardanglingscratchdir on Windows, > includes: > 1. dfs.support.append is set to false in Azure cluster, need an alternative > way when append is disabled > 2. fix for cmd scripts > 3. fix UT on Windows -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13463) Fix ImportSemanticAnalyzer to allow for different src/dst filesystems
[ https://issues.apache.org/jira/browse/HIVE-13463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zach York updated HIVE-13463: - Attachment: HIVE-13463-4.patch Looks like jenkins failed to pick up the last patch. Reattaching > Fix ImportSemanticAnalyzer to allow for different src/dst filesystems > - > > Key: HIVE-13463 > URL: https://issues.apache.org/jira/browse/HIVE-13463 > Project: Hive > Issue Type: Bug > Components: Import/Export >Affects Versions: 2.0.0 >Reporter: Zach York >Assignee: Zach York > Attachments: HIVE-13463-1.patch, HIVE-13463-2.patch, > HIVE-13463-3.patch, HIVE-13463-4.patch, HIVE-13463.patch > > > In ImportSemanticAnalyzer, there is an assumption that the src filesystem for > import and the final location are on the same filesystem. Therefore the check > for emptiness and getExternalTmpLocation will be looking on the wrong > filesystem and will cause an error. The output path should be fed into > getExternalTmpLocation to get a temporary file on the correct filesystem. The > check for emptiness should use the output filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13467) Show llap info on hs2 ui when available
[ https://issues.apache.org/jira/browse/HIVE-13467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254916#comment-15254916 ] Vikram Dixit K commented on HIVE-13467: --- +1. LGTM. > Show llap info on hs2 ui when available > --- > > Key: HIVE-13467 > URL: https://issues.apache.org/jira/browse/HIVE-13467 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-13467.1.patch, HIVE-13467.2.patch, > HIVE-13467.3.patch, HIVE-13467.4.patch, HIVE-13467.5.patch, > HIVE-13467.6.patch, screen-shot-llap.png, screen.png > > > When llap is on and hs2 is configured with access to an llap cluster, HS2 UI > should show some status of the daemons and provide a mechanism to click > through to their respective UIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13595) HiveServer2: Evaluate if ThriftJDBCBinarySerde should implement VectorizedSerde
[ https://issues.apache.org/jira/browse/HIVE-13595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13595: Issue Type: Sub-task (was: Bug) Parent: HIVE-12427 > HiveServer2: Evaluate if ThriftJDBCBinarySerde should implement > VectorizedSerde > --- > > Key: HIVE-13595 > URL: https://issues.apache.org/jira/browse/HIVE-13595 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2 >Affects Versions: 2.1.0 >Reporter: Vaibhav Gumashta > > As part of HIVE-12049, ThriftJDBCBinarySerde was introduced which buffers > rows and writes thrift converted columnar row batches as part of the final > task output. Hive has VectorizedSerde which is used during vectorized > operations. We should explore if ThriftJDBCBinarySerde should implement that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254908#comment-15254908 ] Ashutosh Chauhan commented on HIVE-13591: - +1 > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254900#comment-15254900 ] Prasanth Jayachandran commented on HIVE-13563: -- I would prefer scaling down the default 256KB buffer size to 64KB. [~owen.omalley] Thoughts? > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254895#comment-15254895 ] Prasanth Jayachandran commented on HIVE-13563: -- 256KB is the default buffer size for base which will make 64KB for delta files which I think is huge when multiple delta files are open. Either we have to scale down the default buffer size to 64KB or change the ratio so that delta files gets 16KB which was previous default. > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13563: - Attachment: HIVE-13563.1.patch [~prasanth_j] [~owen.omalley] Can you please review? > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13563: - Status: Patch Available (was: Open) > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > Attachments: HIVE-13563.1.patch > > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13512) Make initializing dag ids in TezWork thread safe for parallel compilation
[
https://issues.apache.org/jira/browse/HIVE-13512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Slawski updated HIVE-13512:
-
Attachment: HIVE-13512.1.patch
Attaching same patch to trigger tests. The logs for the previous test run are
lost due to the Jenkins server being down.
> Make initializing dag ids in TezWork thread safe for parallel compilation
> -
>
> Key: HIVE-13512
> URL: https://issues.apache.org/jira/browse/HIVE-13512
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Planning
>Affects Versions: 2.0.0
>Reporter: Peter Slawski
>Assignee: Peter Slawski
>Priority: Minor
> Attachments: HIVE-13512.1.patch, HIVE-13512.1.patch
>
>
> When parallel query compilation is enabled, it is possible for concurrent
> running threads to create TezWork objects that have the same dag id. This is
> because the counter used to obtain the next dag id is not thread safe. The
> counter should be an AtomicInteger rather than an int.
> {code:java}
> private static int counter;
> ...
> public TezWork(String queryId, Configuration conf) {
> this.dagId = queryId + ":" + (++counter);
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13232) Aggressively drop compression buffers in ORC OutStreams
[ https://issues.apache.org/jira/browse/HIVE-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254869#comment-15254869 ] Prasanth Jayachandran commented on HIVE-13232: -- Backported to branch-1 and branch-2.0 > Aggressively drop compression buffers in ORC OutStreams > --- > > Key: HIVE-13232 > URL: https://issues.apache.org/jira/browse/HIVE-13232 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 1.3.0, 2.1.0, 2.0.1 > > Attachments: HIVE-13232-branch-1.patch, HIVE-13232.patch, > HIVE-13232.patch, HIVE-13232.patch > > > In Hive 0.11, when ORC's OutStream's were flushed they dropped all of the > their buffers. In the patch for HIVE-4324, we inadvertently changed that > behavior so that one of the buffers is held on to. For queries with a lot of > writers and thus under significant memory pressure this can have a > significant impact on the memory usage. > Note that "hive.optimize.sort.dynamic.partition" avoids this problem by > sorting on the dynamic partition key and thus only a single ORC writer is > open at once. This will use memory more effectively and avoid creating ORC > files with very small stripes, which will produce better downstream > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13232) Aggressively drop compression buffers in ORC OutStreams
[ https://issues.apache.org/jira/browse/HIVE-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13232: - Attachment: HIVE-13232-branch-1.patch > Aggressively drop compression buffers in ORC OutStreams > --- > > Key: HIVE-13232 > URL: https://issues.apache.org/jira/browse/HIVE-13232 > Project: Hive > Issue Type: Bug > Components: ORC >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 2.1.0 > > Attachments: HIVE-13232-branch-1.patch, HIVE-13232.patch, > HIVE-13232.patch, HIVE-13232.patch > > > In Hive 0.11, when ORC's OutStream's were flushed they dropped all of the > their buffers. In the patch for HIVE-4324, we inadvertently changed that > behavior so that one of the buffers is held on to. For queries with a lot of > writers and thus under significant memory pressure this can have a > significant impact on the memory usage. > Note that "hive.optimize.sort.dynamic.partition" avoids this problem by > sorting on the dynamic partition key and thus only a single ORC writer is > open at once. This will use memory more effectively and avoid creating ORC > files with very small stripes, which will produce better downstream > performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254860#comment-15254860 ] Thejas M Nair commented on HIVE-13591: -- Found out the reason why it was failing for me even with older commits. I had a $HOME/.beeline/beeline.properties file which was interfering with the test run! Deleting that helped to get it working with older commits. > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13563) Hive Streaming does not honor orc.compress.size and orc.stripe.size table properties
[ https://issues.apache.org/jira/browse/HIVE-13563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13563: - Labels: TODOC2.1 (was: ) > Hive Streaming does not honor orc.compress.size and orc.stripe.size table > properties > > > Key: HIVE-13563 > URL: https://issues.apache.org/jira/browse/HIVE-13563 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Labels: TODOC2.1 > > According to the doc: > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ORC#LanguageManualORC-HiveQLSyntax > One should be able to specify tblproperties for many ORC options. > But the settings for orc.compress.size and orc.stripe.size don't take effect. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some queries with Union all fail when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Summary: Some queries with Union all fail when CBO is off (was: Some
queries with Union all fails when CBO is off)
> Some queries with Union all fail when CBO is off
>
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some queries with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Summary: Some queries with Union all fails when CBO is off (was: Some
query with Union all fails when CBO is off)
> Some queries with Union all fails when CBO is off
> -
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13591: - Description: Not sure at what point this started to fail. was:Not sure at what point this started to fail. > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13591: - Status: Patch Available (was: Open) > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13591: - Attachment: HIVE-13591.1.patch [~thejas] / [~ashutoshc] This seems to be because of a missing parenthesis in the init script introduced by my previous change. Its a very minor patch to review. Thanks Hari > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13591.1.patch > > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan reassigned HIVE-13591: Assignee: Hari Sankar Sivarama Subramaniyan > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan >Assignee: Hari Sankar Sivarama Subramaniyan > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13447) LLAP: check ZK acls for registry and fail if they are too permissive
[
https://issues.apache.org/jira/browse/HIVE-13447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13447:
Attachment: HIVE-13447.01.patch
Added the no-namespace call. Thanks for the pointer!
> LLAP: check ZK acls for registry and fail if they are too permissive
>
>
> Key: HIVE-13447
> URL: https://issues.apache.org/jira/browse/HIVE-13447
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13447.01.patch, HIVE-13447.patch
>
>
> Only the current ("hive") user can have write access.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13424) Refactoring the code to pass a QueryState object rather than HiveConf object
[ https://issues.apache.org/jira/browse/HIVE-13424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254756#comment-15254756 ] Sergey Shelukhin commented on HIVE-13424: - +1 > Refactoring the code to pass a QueryState object rather than HiveConf object > > > Key: HIVE-13424 > URL: https://issues.apache.org/jira/browse/HIVE-13424 > Project: Hive > Issue Type: Sub-task > Components: Query Processor >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-13424.1.patch, HIVE-13424.2.patch, > HIVE-13424.3.patch, HIVE-13424.4.patch, HIVE-13424.5.patch > > > Step1: to refractor the code by creating the QueryState class and moving > query related info from SessionState. Then during compilation, execution > stages, pass single QueryState object for each query. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-13594) Misc cleanup on llap branch
[ https://issues.apache.org/jira/browse/HIVE-13594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere resolved HIVE-13594. --- Resolution: Fixed Fix Version/s: llap Committed to llap branch > Misc cleanup on llap branch > --- > > Key: HIVE-13594 > URL: https://issues.apache.org/jira/browse/HIVE-13594 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > Attachments: HIVE-13594.1.patch > > > - Remove unnecessary config settings added to llap branch > - Revert unnecessary changes to pom/source files. > - Rework HiveSplitGenerator changes per comments in that file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254738#comment-15254738 ] Wei Zheng commented on HIVE-12837: -- [~sershe] Could you please review? > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13594) Misc cleanup on llap branch
[ https://issues.apache.org/jira/browse/HIVE-13594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13594: -- Attachment: HIVE-13594.1.patch > Misc cleanup on llap branch > --- > > Key: HIVE-13594 > URL: https://issues.apache.org/jira/browse/HIVE-13594 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13594.1.patch > > > - Remove unnecessary config settings added to llap branch > - Revert unnecessary changes to pom/source files. > - Rework HiveSplitGenerator changes per comments in that file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12837) Better memory estimation/allocation for hybrid grace hash join during hash table loading
[ https://issues.apache.org/jira/browse/HIVE-12837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-12837: - Attachment: HIVE-12837.4.patch Rebased patch 4 for test > Better memory estimation/allocation for hybrid grace hash join during hash > table loading > > > Key: HIVE-12837 > URL: https://issues.apache.org/jira/browse/HIVE-12837 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.1.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-12837.1.patch, HIVE-12837.2.patch, > HIVE-12837.3.patch, HIVE-12837.4.patch > > > This is to avoid an edge case when the memory available is very little (less > than a single write buffer size), and we start loading the hash table. Since > the write buffer is lazily allocated, we will easily run out of memory before > even checking if we should spill any hash partition. > e.g. > Total memory available: 210 MB > Size of ref array of BytesBytesMultiHashMap for each hash partition: ~16 MB > Size of write buffer: 8 MB (lazy allocation) > Number of hash partitions: 16 > Number of hash partitions created in memory: 13 > Number of hash partitions created on disk: 3 > Available memory left after HybridHashTableContainer initialization: > 210-16*13=2MB > Now let's say a row is to be loaded into a hash partition in memory, it will > try to allocate an 8MB write buffer for it, but we only have 2MB, thus OOM. > Solution is to perform the check for possible spilling earlier so we can > spill partitions if memory is about to be full, to avoid OOM. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12079) Add units tests for HiveServer2 LDAP filters added in HIVE-7193
[ https://issues.apache.org/jira/browse/HIVE-12079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254711#comment-15254711 ] Naveen Gangam commented on HIVE-12079: -- The test failures are not related to the change. The change only involves changes to test code. All new tests that have been added have passed. +1 for me. > Add units tests for HiveServer2 LDAP filters added in HIVE-7193 > --- > > Key: HIVE-12079 > URL: https://issues.apache.org/jira/browse/HIVE-12079 > Project: Hive > Issue Type: Test > Components: HiveServer2 >Affects Versions: 1.1.1 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-12079.patch > > > HIVE-11866 adds a test framework that uses an in-memory ldap server for unit > tests. Need to add unit tests for user and group filtering feature added in > HIVE-7193. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Attachment: HIVE-13570.2.patch
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254706#comment-15254706
]
Yongzhi Chen commented on HIVE-13570:
-
The failures seem not related. Most failures except the Llap tests have age of
3.
I run some Llap tests may related to the fix:
tez_union.q,tez_union_multiinsert.q,tez_union2.q , all pass:
{noformat}
Running org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 90.089 sec - in
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
{noformat}
I will remove and reattach patch 2 to run the unit tests again.
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13570:
Attachment: (was: HIVE-13570.2.patch)
> Some query with Union all fails when CBO is off
> ---
>
> Key: HIVE-13570
> URL: https://issues.apache.org/jira/browse/HIVE-13570
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13570.1.PATCH, HIVE-13570.2.patch
>
>
> Some queries with union all throws IndexOutOfBoundsException
> when:
> set hive.cbo.enable=false;
> set hive.ppd.remove.duplicatefilters=true;
> The stack is as:
> {noformat}
> java.lang.IndexOutOfBoundsException: Index: 67, Size: 67
> at java.util.ArrayList.rangeCheck(ArrayList.java:635)
> at java.util.ArrayList.get(ArrayList.java:411)
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.genColLists(ColumnPrunerProcCtx.java:161)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcCtx.handleFilterUnionChildren(ColumnPrunerProcCtx.java:273)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPrunerProcFactory$ColumnPrunerFilterProc.process(ColumnPrunerProcFactory.java:108)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner$ColumnPrunerWalker.walk(ColumnPruner.java:172)
>
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>
> at
> org.apache.hadoop.hive.ql.optimizer.ColumnPruner.transform(ColumnPruner.java:135)
>
> at
> org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:198)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10327)
>
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:192)
>
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
>
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:432)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1119)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1167)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1055)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1045)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
> at
> org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
> at
> org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12079) Add units tests for HiveServer2 LDAP filters added in HIVE-7193
[
https://issues.apache.org/jira/browse/HIVE-12079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254693#comment-15254693
]
Hive QA commented on HIVE-12079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800240/HIVE-12079.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 39 failed/errored test(s), 9948 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.beeline.TestSchemaTool.testSchemaInit
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.org.apache.hive.service.TestHS2ImpersonationWithRemoteMS
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/42/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/42/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-42/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 39 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12800240 - PreCommit-HIVE-MASTER-Build
> Add units
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254674#comment-15254674 ] Ashutosh Chauhan commented on HIVE-13591: - [~thejas] your two comments are contradictory : ) HIVE-13349 can't make it fail on Mar 22 trunk. Further, I remember this test being flaky for a while now. > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254669#comment-15254669 ] Sushanth Sowmyan commented on HIVE-13480: - Thanks, committed to master. > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan updated HIVE-13480: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Fix For: 2.1.0 > > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254655#comment-15254655
]
Pengcheng Xiong commented on HIVE-7693:
---
the failures still exist. We did not figure out a way to fix them. :(
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch,
> HIVE-7693.06.patch, HIVE-7693.07.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13591) TestSchemaTool is failing on master
[ https://issues.apache.org/jira/browse/HIVE-13591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254654#comment-15254654 ] Thejas M Nair commented on HIVE-13591: -- My suspicion is that HIVE-13349 is causing the problem. It is complaining about the derby initialization script and that is the only patch that changed it recently. Also this test failed in the test run for the jira and has been seen since its commit. [~hsubramaniyan] can you please take a look ? But there is one thing weird, it is failing for me even when I go back all the way to 22 nd March commits in master. Seems like something gets left behind . > TestSchemaTool is failing on master > --- > > Key: HIVE-13591 > URL: https://issues.apache.org/jira/browse/HIVE-13591 > Project: Hive > Issue Type: Test > Components: Metastore >Affects Versions: 2.1.0 >Reporter: Ashutosh Chauhan > > Not sure at what point this started to fail. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254648#comment-15254648
]
Ashutosh Chauhan commented on HIVE-7693:
thats what I meant are those q test failures still exist. If so, is there way
to fix them?
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch,
> HIVE-7693.06.patch, HIVE-7693.07.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13480) Add hadoop2 metrics reporter for Codahale metrics
[ https://issues.apache.org/jira/browse/HIVE-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254645#comment-15254645 ] Thejas M Nair commented on HIVE-13480: -- +1 > Add hadoop2 metrics reporter for Codahale metrics > - > > Key: HIVE-13480 > URL: https://issues.apache.org/jira/browse/HIVE-13480 > Project: Hive > Issue Type: Bug >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-13480.2.patch, HIVE-13480.3.patch, > HIVE-13480.4.patch, HIVE-13480.5.patch, HIVE-13480.patch > > > Multiple other apache components allow sending metrics over to Hadoop2 > metrics, which allow for monitoring solutions like Ambari Metrics Server to > work against that to show metrics for components in one place. Our Codahale > metrics works very well, so ideally, we would like to bridge the two, to > allow Codahale to add a Hadoop2 reporter that enables us to continue to use > Codahale metrics (i.e. not write another custom metrics impl) but report > using Hadoop2. > Apache Phoenix also had such a recent usecase and were in the process of > adding in a stub piece that allows this forwarding. We should use the same > reporter to minimize redundancy while pushing metrics to a centralized > solution like Hadoop2 Metrics/AMS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254643#comment-15254643
]
Pengcheng Xiong commented on HIVE-7693:
---
It solved this issue but broke other q tests
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch,
> HIVE-7693.06.patch, HIVE-7693.07.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254637#comment-15254637
]
Ashutosh Chauhan commented on HIVE-7693:
[~pxiong] Can you rebase your patch for this one? Do these previously
unresolved issues still remain?
> Invalid column ref error in order by when using column alias in select clause
> and using having
> --
>
> Key: HIVE-7693
> URL: https://issues.apache.org/jira/browse/HIVE-7693
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 0.13.0
>Reporter: Deepesh Khandelwal
>Assignee: Pengcheng Xiong
> Attachments: HIVE-7693.01.patch, HIVE-7693.02.patch,
> HIVE-7693.03.patch, HIVE-7693.04.patch, HIVE-7693.05.patch,
> HIVE-7693.06.patch, HIVE-7693.07.patch
>
>
> Hive CLI session:
> {noformat}
> hive> create table abc(foo int, bar string);
> OK
> Time taken: 0.633 seconds
> hive> select foo as c0, count(*) as c1 from abc group by foo, bar having bar
> like '%abc%' order by foo;
> FAILED: SemanticException [Error 10004]: Line 1:93 Invalid table alias or
> column reference 'foo': (possible column names are: c0, c1)
> {noformat}
> Without having clause, the query runs fine, example:
> {code}
> select foo as c0, count(*) as c1 from abc group by foo, bar order by foo;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13346) LLAP doesn't update metadata priority when reusing from cache; some tweaks in LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254610#comment-15254610 ] Sergey Shelukhin commented on HIVE-13346: - [~prasanth_j] can you review? > LLAP doesn't update metadata priority when reusing from cache; some tweaks in > LRFU policy > - > > Key: HIVE-13346 > URL: https://issues.apache.org/jira/browse/HIVE-13346 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13346.01.patch, HIVE-13346.02.patch, > HIVE-13346.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13346) LLAP doesn't update metadata priority when reusing from cache; some tweaks in LRFU policy
[ https://issues.apache.org/jira/browse/HIVE-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13346: Attachment: HIVE-13346.02.patch Removed some of the change, added checking to the test for the original issue (priorities not updated). > LLAP doesn't update metadata priority when reusing from cache; some tweaks in > LRFU policy > - > > Key: HIVE-13346 > URL: https://issues.apache.org/jira/browse/HIVE-13346 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13346.01.patch, HIVE-13346.02.patch, > HIVE-13346.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13510) Dynamic partitioning doesn’t work when remote metastore is used
[
https://issues.apache.org/jira/browse/HIVE-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Illya Yalovyy updated HIVE-13510:
-
Status: Patch Available (was: Open)
> Dynamic partitioning doesn’t work when remote metastore is used
> ---
>
> Key: HIVE-13510
> URL: https://issues.apache.org/jira/browse/HIVE-13510
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: Hadoop 2.7.1
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
>Priority: Critical
> Attachments: HIVE-13510.1.patch
>
>
> *Steps to reproduce:*
> # Configure remote metastore (hive.metastore.uris)
> # Create table t1 (a string);
> # Create table t2 (a string) partitioned by (b string);
> # set hive.exec.dynamic.partition.mode=nonstrict;
> # Insert overwrite table t2 partition (b) select a,a from t1;
> *Result:*
> {noformat}
> FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> 16/04/13 15:04:51 [c679e424-2501-4347-8146-cf1b1cae217c main]: ERROR
> ql.Driver: FAILED: SemanticException
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> org.apache.hadoop.hive.ql.parse.SemanticException:
> org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> at
> org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:84)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genFileSinkPlan(SemanticAnalyzer.java:6550)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9315)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9204)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10071)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9949)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10607)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:358)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10618)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:233)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:245)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:476)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:318)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1192)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1287)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1118)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1106)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:236)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:187)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:339)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:748)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:721)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:648)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.thrift.TApplicationException: getMetaConf failed: unknown result
> at org.apache.hadoop.hive.ql.metadata.Hive.getMetaConf(Hive.java:3493)
> at
> org.apache.hadoop.hive.ql.plan.DynamicPartitionCtx.(DynamicPartitionCtx.java:82)
> ... 29 more
> Caused by: org.apache.thrift.TApplicationException: getMetaConf failed:
> unknown result
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_getMetaConf(ThriftHiveMetastore.java:666)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.getMetaConf(ThriftHiveMetastore.java:646)
> at
>
[jira] [Commented] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254590#comment-15254590
]
Rohit Dholakia commented on HIVE-12049:
---
Welcome, it's been great working on the patch and related work! Thanks
[~vgumashta] [~thejas] for all the help!
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13585) Add counter metric for direct sql failures
[ https://issues.apache.org/jira/browse/HIVE-13585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254568#comment-15254568 ] Sergey Shelukhin commented on HIVE-13585: - +1 > Add counter metric for direct sql failures > -- > > Key: HIVE-13585 > URL: https://issues.apache.org/jira/browse/HIVE-13585 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-13585.patch > > > In case of direct sql failure, metastore query falls back to DataNucleus. > It'd be good to record how often this happens as a metrics counter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Labels: TODOC2.1 (was: )
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Labels: TODOC2.1
> Fix For: 2.1.0
>
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13592) metastore calls map is not thread safe
[ https://issues.apache.org/jira/browse/HIVE-13592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254522#comment-15254522 ] Sergey Shelukhin commented on HIVE-13592: - No. There's no sync between get and put. replace() acts as CAS, and putIfAbsent() is just a special case of that. Otherwise, nothing prevents T1-get, T2-get, T2-put, T1-put from happening, and losing the values. > metastore calls map is not thread safe > -- > > Key: HIVE-13592 > URL: https://issues.apache.org/jira/browse/HIVE-13592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13592.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10438) HiveServer2: Enable ResultSet compression for HS2/*DBC driver
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10438: Summary: HiveServer2: Enable ResultSet compression for HS2/*DBC driver (was: Enable ResultSet compression for HS2/*DBC driver) > HiveServer2: Enable ResultSet compression for HS2/*DBC driver > - > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Rohit Dholakia > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254512#comment-15254512 ] Jesus Camacho Rodriguez commented on HIVE-13068: I correct myself: I think it will make a performance difference if we fold the CAST in SemanticAnalyzer, as we can perform it as we create the expressions... I will try to move the logic there and submit a new patch. I will explore further the other two issues. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12428) HiveServer2: Provide an option for HiveServer2 to stream serialized thrift results when they are available
[ https://issues.apache.org/jira/browse/HIVE-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12428: Summary: HiveServer2: Provide an option for HiveServer2 to stream serialized thrift results when they are available (was: Provide an option for HiveServer2 to stream serialized thrift results when they are available) > HiveServer2: Provide an option for HiveServer2 to stream serialized thrift > results when they are available > -- > > Key: HIVE-12428 > URL: https://issues.apache.org/jira/browse/HIVE-12428 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Once HIVE-12049 is complete, we will have an option on HS2 side to avoid > paying the deserialization penalty and simple stream batches of rows on the > wire which the *DBC driver can appropriately interpret. This can be done by > changing the FetchOperator such that when it has the thrift formatted row > sets available, it uses the fastpath. For queries where tasks don't write > thrift rowsets, the current implementation will suffice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12428) HiveServer2: Provide an option for HiveServer2 to stream serialized thrift results when they are available
[ https://issues.apache.org/jira/browse/HIVE-12428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12428: Affects Version/s: 2.0.0 > HiveServer2: Provide an option for HiveServer2 to stream serialized thrift > results when they are available > -- > > Key: HIVE-12428 > URL: https://issues.apache.org/jira/browse/HIVE-12428 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > > Once HIVE-12049 is complete, we will have an option on HS2 side to avoid > paying the deserialization penalty and simple stream batches of rows on the > wire which the *DBC driver can appropriately interpret. This can be done by > changing the FetchOperator such that when it has the thrift formatted row > sets available, it uses the fastpath. For queries where tasks don't write > thrift rowsets, the current implementation will suffice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12442) HiveServer2: Refactor/repackage HiveServer2's Thrift code so that it can be used in the tasks
[ https://issues.apache.org/jira/browse/HIVE-12442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12442: Affects Version/s: (was: 1.2.1) 2.0.0 > HiveServer2: Refactor/repackage HiveServer2's Thrift code so that it can be > used in the tasks > - > > Key: HIVE-12442 > URL: https://issues.apache.org/jira/browse/HIVE-12442 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta >Assignee: Rohit Dholakia > Labels: HiveServer2, RPC, Thrift > Fix For: 2.1.0 > > Attachments: hive-12442.1.patch, hive-12442.2.patch, > hive-12442.3.patch, hive-12442.4.patch, hive-12442.5.patch > > > For implementing HIVE-12427, the tasks will need to have knowledge of thrift > types from HS2's thrift API. This jira will look at the least invasive way to > do that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12427) HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance
[ https://issues.apache.org/jira/browse/HIVE-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-12427: Summary: HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance (was: Improve HiveServer2 *DBC ResultSet performance) > HiveServer2: Improve HiveServer2 JDBC/ODBC ResultSet performance > > > Key: HIVE-12427 > URL: https://issues.apache.org/jira/browse/HIVE-12427 > Project: Hive > Issue Type: Improvement > Components: HiveServer2, JDBC, ODBC >Reporter: Vaibhav Gumashta > > The current implementation does a lot of CPU intensive work in HiveServer2. > It deserializes the temporary results written to disk and also serializes > the results into thrift objects in HS2. This adds to the latency of fetching > results via HS2. Doing that work in the task nodes would get that work done > in parallel and scale it, thereby reducing the time it takes to retrieve > large results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13593) HiveServer2: Better performance instrumentation for HIVE-12049 (serializing thrift ResultSets in tasks)
[ https://issues.apache.org/jira/browse/HIVE-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13593: Issue Type: Sub-task (was: Bug) Parent: HIVE-12427 > HiveServer2: Better performance instrumentation for HIVE-12049 (serializing > thrift ResultSets in tasks) > --- > > Key: HIVE-13593 > URL: https://issues.apache.org/jira/browse/HIVE-13593 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Affects Versions: 2.0.0 >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[
https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254505#comment-15254505
]
Jesus Camacho Rodriguez commented on HIVE-13068:
I agree with you.
However, I think it is difficult because of the current limitations that I
outlined above.
- For instance, is there a way to specify that a given constant should be of a
certain type in HQL without using cast/udfto? I think there is not, that is why
I came with this solution.
I could try to add the logic to fold the expression to the SemanticAnalyzer,
thus not traversing the tree. But I think it will not make such a big
difference performance wise and we will end up adding more logic to
SemanticAnalyzer, which is already huge. Further, if return path is in place at
some point, it will be more difficult to detect/remove that unnecessary logic
too.
- The problem with {{power}} expression is that it was returning NaN when it
was folded, and I think we cannot convert that currently through AST? It was a
corner case and I did not explore it much further as the patch was so full of
small pieces. But I will explore it further.
- The filter removal for {{true}} predicates comes practically for free if we
need to execute SimpleConstantReduction, but I agree I could try to rewrite
that logic in SemanticAnalyzer.
The fact that we are not carrying constants all over the operator tree and
checking all the expressions to replace them, should make a performance
difference already. I agree with you that we need to take it a step further,
but without further integration work, I did not figure out a better solution
for some of these cases (specially the CAST, that is quite common because of
Calcite tight type system).
> Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
> ---
>
> Key: HIVE-13068
> URL: https://issues.apache.org/jira/browse/HIVE-13068
> Project: Hive
> Issue Type: Sub-task
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch,
> HIVE-13068.patch
>
>
> After HIVE-12543 went in, we need follow-up work to disable the last call to
> ConstantPropagate in Hive. This probably implies work on extending the
> constant folding logic in Calcite.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.0
Target Version/s: 2.1.0
Status: Resolved (was: Patch Available)
Patch committed to master. Thanks [~rohitdholakia] for the work!
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Fix For: 2.1.0
>
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12049) HiveServer2: Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-12049:
Summary: HiveServer2: Provide an option to write serialized thrift objects
in final tasks (was: Provide an option to write serialized thrift objects in
final tasks)
> HiveServer2: Provide an option to write serialized thrift objects in final
> tasks
>
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13068) Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II
[ https://issues.apache.org/jira/browse/HIVE-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254475#comment-15254475 ] Ashutosh Chauhan commented on HIVE-13068: - Historically latency of doing optimization on Hive tree comes from slowness of traversing operator & expression tree, not of doing optimization itself (executing folding logic in this case). So my guess is even if you are executing very little logic latency of doing walk on trees will outnumber any gains. So, my assumption is we will see benefit of migration only if we can totally avoid the optimization on Hive tree. Further, it still leaves the room for bugs in new code, which is avoidable. So my suggestion is to instead try to fix all issues which comes out of it, instead of having this new simplified rule. > Disable Hive ConstantPropagate optimizer when CBO has optimized the plan II > --- > > Key: HIVE-13068 > URL: https://issues.apache.org/jira/browse/HIVE-13068 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13068.01.patch, HIVE-13068.01.patch, > HIVE-13068.patch > > > After HIVE-12543 went in, we need follow-up work to disable the last call to > ConstantPropagate in Hive. This probably implies work on extending the > constant folding logic in Calcite. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13523) Fix connection leak in ORC RecordReader and refactor for unit testing
[ https://issues.apache.org/jira/browse/HIVE-13523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254464#comment-15254464 ] Owen O'Malley commented on HIVE-13523: -- I've redone this in my HIVE-12159 patch. The RecordReaderImpl now takes ReaderImpl and Reader.Options. It is simple and direct. > Fix connection leak in ORC RecordReader and refactor for unit testing > - > > Key: HIVE-13523 > URL: https://issues.apache.org/jira/browse/HIVE-13523 > Project: Hive > Issue Type: Bug > Components: ORC >Affects Versions: 2.0.0 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Fix For: 2.1.0, 2.0.1 > > Attachments: HIVE-13523.patch > > > In RecordReaderImpl, a MetadataReaderImpl object was being created (opening a > file), but never closed, causing a leak. This change closes the Metadata > object in RecordReaderImpl, and does substantial refactoring to make > RecordReaderImpl testable: > * Created DataReaderFactory and MetadataReaderFactory (plus default > implementations) so that the create() methods can be mocked to verify that > the objects are actually closed in RecordReaderImpl.close() > * Created MetadataReaderProperties and DataReaderProperties to clean up > argument lists, making code more readable > * Created a builder() for RecordReaderImpl to make the code more readable > * DataReader and MetadataReader now extend closeable (there was no reason > for them not to in the first place) so I can use the guava Closer interface: > http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/io/Closer.html > * Use the Closer interface to guarantee that regardless of if either close() > call fails, both will be attempted (preventing further potential leaks) > * Create builders for MetadataReaderProperties, DataReaderProperties, and > RecordReaderImpl to help with code readability -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11887) spark tests break the build on a shared machine, can break HiveQA
[ https://issues.apache.org/jira/browse/HIVE-11887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-11887: Attachment: HIVE-11887.02.patch Same patch. I never have time to drive this to conclusion... > spark tests break the build on a shared machine, can break HiveQA > - > > Key: HIVE-11887 > URL: https://issues.apache.org/jira/browse/HIVE-11887 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-11887.01.patch, HIVE-11887.02.patch, > HIVE-11887.patch > > > Spark download creates UDFExampleAdd jar in /tmp; when building on a shared > machine, someone else's jar from a build prevents this jar from being created > (I have no permissions to this file because it was created by a different > user) and the build fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13221) expose metastore APIs from HS2
[ https://issues.apache.org/jira/browse/HIVE-13221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254461#comment-15254461 ] Sergey Shelukhin commented on HIVE-13221: - [~thejas] [~sushanth] ping? > expose metastore APIs from HS2 > -- > > Key: HIVE-13221 > URL: https://issues.apache.org/jira/browse/HIVE-13221 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13221.01.patch, HIVE-13221.patch > > > I was always wondering why we don't do that, for the people who run HS2 and > also need metastore due to it being used externally; they don't need to run a > standalone metastore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[ https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13391: Attachment: HIVE-13391.05.patch The same patch for QA... Grrr > add an option to LLAP to use keytab to authenticate to read data > > > Key: HIVE-13391 > URL: https://issues.apache.org/jira/browse/HIVE-13391 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, > HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, > HIVE-13391.patch > > > This can be used for non-doAs case to allow access to clients who don't > propagate HDFS tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13449: Attachment: HIVE-13449.01.patch Resubmitting the same patch for HiveQA > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.01.patch, HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13449) LLAP: HS2 should get the token directly, rather than from LLAP
[ https://issues.apache.org/jira/browse/HIVE-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254439#comment-15254439 ] Sergey Shelukhin commented on HIVE-13449: - [~vikram.dixit] [~hagleitn] perhaps you can review this > LLAP: HS2 should get the token directly, rather than from LLAP > -- > > Key: HIVE-13449 > URL: https://issues.apache.org/jira/browse/HIVE-13449 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13449.patch > > > HS2 doesn't need a roundtrip to LLAP; it can instantiate the SecretManager > directly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12049) Provide an option to write serialized thrift objects in final tasks
[
https://issues.apache.org/jira/browse/HIVE-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254434#comment-15254434
]
Vaibhav Gumashta commented on HIVE-12049:
-
Test failures look unrelated - I'll commit shortly.
> Provide an option to write serialized thrift objects in final tasks
> ---
>
> Key: HIVE-12049
> URL: https://issues.apache.org/jira/browse/HIVE-12049
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Rohit Dholakia
>Assignee: Rohit Dholakia
> Attachments: HIVE-12049.1.patch, HIVE-12049.11.patch,
> HIVE-12049.12.patch, HIVE-12049.13.patch, HIVE-12049.14.patch,
> HIVE-12049.15.patch, HIVE-12049.16.patch, HIVE-12049.17.patch,
> HIVE-12049.18.patch, HIVE-12049.19.patch, HIVE-12049.2.patch,
> HIVE-12049.25.patch, HIVE-12049.26.patch, HIVE-12049.3.patch,
> HIVE-12049.4.patch, HIVE-12049.5.patch, HIVE-12049.6.patch,
> HIVE-12049.7.patch, HIVE-12049.9.patch, new-driver-profiles.png,
> old-driver-profiles.png
>
>
> For each fetch request to HiveServer2, we pay the penalty of deserializing
> the row objects and translating them into a different representation suitable
> for the RPC transfer. In a moderate to high concurrency scenarios, this can
> result in significant CPU and memory wastage. By having each task write the
> appropriate thrift objects to the output files, HiveServer2 can simply stream
> a batch of rows on the wire without incurring any of the additional cost of
> deserialization and translation.
> This can be implemented by writing a new SerDe, which the FileSinkOperator
> can use to write thrift formatted row batches to the output file. Using the
> pluggable property of the {{hive.query.result.fileformat}}, we can set it to
> use SequenceFile and write a batch of thrift formatted rows as a value blob.
> The FetchTask can now simply read the blob and send it over the wire. On the
> client side, the *DBC driver can read the blob and since it is already
> formatted in the way it expects, it can continue building the ResultSet the
> way it does in the current implementation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13570) Some query with Union all fails when CBO is off
[
https://issues.apache.org/jira/browse/HIVE-13570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15254435#comment-15254435
]
Hive QA commented on HIVE-13570:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12800229/HIVE-13570.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 71 failed/errored test(s), 9939 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_bucket_map_join_tez2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_3
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_4
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_cte_mat_5
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_hybridgrace_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llapdecider
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dml
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_hash
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_join_tests
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_joins_explain
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_smb_main
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_union_multiinsert
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_1
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_tez_vector_dynpart_hashjoin_2
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRemoteUGIHiveMetaStoreIpAddress.testIpAddress
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.lockConflictDbTable
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
