[jira] [Commented] (HIVE-13670) Improve Beeline reconnect semantics
[
https://issues.apache.org/jira/browse/HIVE-13670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15270111#comment-15270111
]
Hive QA commented on HIVE-13670:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801804/HIVE-13670.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 10002 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-smb_cache.q-transform_ppr2.q-vector_outer_join0.q-and-3-more
- did not produce a TEST-*.xml file
TestMiniTezCliDriver-vector_coalesce.q-cbo_windowing.q-tez_join.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cross_product_check_1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cte_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynpart_sort_optimization
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_parallel
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_script_env_var2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_transform_ppr1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_if_expr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_nested_udf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_parquet
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.exec.tez.TestDynamicPartitionPruner.testSingleSourceMultipleFiltersOrdering1
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager.testLockTimeout
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
[jira] [Updated] (HIVE-13425) Fix partition addition in MSCK REPAIR TABLE command
[
https://issues.apache.org/jira/browse/HIVE-13425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shinichi Yamashita updated HIVE-13425:
--
Attachment: HIVE-13425.2.patch
> Fix partition addition in MSCK REPAIR TABLE command
> ---
>
> Key: HIVE-13425
> URL: https://issues.apache.org/jira/browse/HIVE-13425
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.0.0
>Reporter: Shinichi Yamashita
>Assignee: Shinichi Yamashita
>Priority: Minor
> Attachments: HIVE-13425.1.patch, HIVE-13425.2.patch
>
>
> I create a table in following HiveQL.
> {code}
> hive> create table example (name string) partitioned by (id int);
> {code}
> And, I make some directories about this table in HDFS.
> {code}
> [root@hadoop ~]# hdfs dfs -ls -R /user/hive/warehouse/example
> drwxr-xr-x - root hadoop 0 2016-04-05 22:21
> /user/hive/warehouse/example/id=1
> drwxr-xr-x - root hadoop 0 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2
> -rw-r--r-- 1 root hadoop 8 2016-04-05 22:22
> /user/hive/warehouse/example/id=1/id=2/example.txt
> {code}
> Next I executed MSCK REPAIR TABLE command and added a partition. And this
> result became as follows.
> {code}
> [root@hadoop ~]# hive -e 'msck repair table example'
> OK
> Partitions not in metastore: example:id=1/id=2
> Repair: Added partition to metastore example:id=1/id=2
> Time taken: 1.243 seconds, Fetched: 2 row(s)
> [root@hadoop ~]# hive -e 'show partitions example'
> OK
> id=2
> {code}
> "id=1" should be a partition, but "id=2" was added. I will fix this problem.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13027) Configuration changes to improve logging performance
[ https://issues.apache.org/jira/browse/HIVE-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13027: - Attachment: HIVE-13027.4.patch Some properties like append, immediateFlush and includeLocation are being set to the defaults. It is redundant. Hence removed in this patch. > Configuration changes to improve logging performance > > > Key: HIVE-13027 > URL: https://issues.apache.org/jira/browse/HIVE-13027 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13027.1.patch, HIVE-13027.2.patch, > HIVE-13027.3.patch, HIVE-13027.3.patch, HIVE-13027.4.patch > > > For LLAP and HS2, some configuration changes can be made to improve logging > performance > 1) LOG4j2's async logger claims to have 6-68 times better performance than > synchronous logger. https://logging.apache.org/log4j/2.x/manual/async.html > 2) Replace File appenders with RandomAccessFileAppender that claims to be > 20-200% more performant. > https://logging.apache.org/log4j/2.x/manual/appenders.html#RandomAccessFileAppender > Also make async logging configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13027) Configuration changes to improve logging performance
[ https://issues.apache.org/jira/browse/HIVE-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-13027: - Attachment: HIVE-13027.3.patch Rebased patch for precommit test run > Configuration changes to improve logging performance > > > Key: HIVE-13027 > URL: https://issues.apache.org/jira/browse/HIVE-13027 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13027.1.patch, HIVE-13027.2.patch, > HIVE-13027.3.patch, HIVE-13027.3.patch > > > For LLAP and HS2, some configuration changes can be made to improve logging > performance > 1) LOG4j2's async logger claims to have 6-68 times better performance than > synchronous logger. https://logging.apache.org/log4j/2.x/manual/async.html > 2) Replace File appenders with RandomAccessFileAppender that claims to be > 20-200% more performant. > https://logging.apache.org/log4j/2.x/manual/appenders.html#RandomAccessFileAppender > Also make async logging configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13443) LLAP: signing for the second state of submit (the event)
[ https://issues.apache.org/jira/browse/HIVE-13443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13443: Summary: LLAP: signing for the second state of submit (the event) (was: LLAP: change JDBC LLAPIF splits to contain protobuf fragment specs (for two stages of submit) ) > LLAP: signing for the second state of submit (the event) > > > Key: HIVE-13443 > URL: https://issues.apache.org/jira/browse/HIVE-13443 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13444) LLAP: add HMAC signatures to LLAP; verify them on LLAP side
[ https://issues.apache.org/jira/browse/HIVE-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13444: Status: Patch Available (was: Open) > LLAP: add HMAC signatures to LLAP; verify them on LLAP side > --- > > Key: HIVE-13444 > URL: https://issues.apache.org/jira/browse/HIVE-13444 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13444.WIP.patch, HIVE-13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13444) LLAP: add HMAC signatures to LLAP; verify them on LLAP side
[ https://issues.apache.org/jira/browse/HIVE-13444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13444: Attachment: HIVE-13444.patch The patch. Signing is disabled for now, I'll enable it for remote API once the local API is committed (some other JIRA). Only the initial request is supported for now. I actually wonder how it would work for Tez protocol for the rest of the splits and whether we should move away from Tez events for this stuff; there's a separate JIRA for adding it to events or equivalent. [~hagleitn] [~sseth] [~jdere] can you review? > LLAP: add HMAC signatures to LLAP; verify them on LLAP side > --- > > Key: HIVE-13444 > URL: https://issues.apache.org/jira/browse/HIVE-13444 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13444.WIP.patch, HIVE-13444.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-7693) Invalid column ref error in order by when using column alias in select clause and using having
[
https://issues.apache.org/jira/browse/HIVE-7693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269987#comment-15269987
]
Hive QA commented on HIVE-7693:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801796/HIVE-7693.08.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 72 failed/errored test(s), 10012 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-order_null.q-vector_acid3.q-orc_merge10.q-and-12-more -
did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_complex_alias
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_minimr_broken_pipe
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query19
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query55
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query71
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.metastore.hbase.TestHBaseImport.org.apache.hadoop.hive.metastore.hbase.TestHBaseImport
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.checkExpectedLocks
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.insertOverwriteCreate
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testDummyTxnManagerOnAcidTable
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLocksInSubquery
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.updateSelectUpdate
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
[jira] [Updated] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13363:
---
Status: Open (was: Patch Available)
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13363:
---
Status: Patch Available (was: Open)
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269979#comment-15269979
]
Anthony Hsu commented on HIVE-13363:
Attached updated patch to fix the build.
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13363) Add hive.metastore.token.signature property to HiveConf
[
https://issues.apache.org/jira/browse/HIVE-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-13363:
---
Attachment: HIVE-13363.2.patch
> Add hive.metastore.token.signature property to HiveConf
> ---
>
> Key: HIVE-13363
> URL: https://issues.apache.org/jira/browse/HIVE-13363
> Project: Hive
> Issue Type: Improvement
>Reporter: Anthony Hsu
>Assignee: Anthony Hsu
> Attachments: HIVE-13363.1.patch, HIVE-13363.2.patch
>
>
> I noticed that the {{hive.metastore.token.signature}} property is not defined
> in HiveConf.java, but hardcoded everywhere it's used in the Hive codebase.
> [HIVE-2963] fixes this but was never committed due to being resolved as a
> duplicate ticket.
> We should add {{hive.metastore.token.signature}} to HiveConf.java to
> centralize its definition and make the property more discoverable (it's
> useful to set it when talking to multiple metastores).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13561) HiveServer2 is leaking ClassLoaders when add jar / temporary functions are used
[ https://issues.apache.org/jira/browse/HIVE-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269966#comment-15269966 ] Vaibhav Gumashta commented on HIVE-13561: - [~tleftwich] Thanks for adding the test! I'll review your patch in a day. > HiveServer2 is leaking ClassLoaders when add jar / temporary functions are > used > --- > > Key: HIVE-13561 > URL: https://issues.apache.org/jira/browse/HIVE-13561 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 1.2.0, 1.2.1, 2.0.0 >Reporter: Trystan Leftwich >Assignee: Trystan Leftwich > Attachments: HIVE-13561-branch-1.2.3.patch, HIVE-13561.2.patch, > HIVE-13561.3.patch > > > I can repo this on branch-1.2 and branch-2.0. > It looks to be the same issues as: HIVE-11408 > The patch from HIVE-11408 looks to fix the issue as well. > I've updated the patch from HIVE-11408 to be aligned with branch-1.2 and > master -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: (was: HIVE-11793.5.patch) > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: HIVE-11793.5.patch > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: HIVE-11793.5.patch Upload patch 5. It deprecated IMetaStoreClient.showLocks() method. > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch, HIVE-11793.5.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13684) Remove the deprecated IMetaStoreClient.showLocks()
[ https://issues.apache.org/jira/browse/HIVE-13684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-13684: - Description: IMetaStoreClient.showLocks() is deprecated in Hive 2.1 via HIVE-11793. This method can be removed in Hive 2.2 release. was: IMetaStoreClient.showLocks() is deprecated in Hive 2.1. This method can be removed in Hive 2.2 release. > Remove the deprecated IMetaStoreClient.showLocks() > -- > > Key: HIVE-13684 > URL: https://issues.apache.org/jira/browse/HIVE-13684 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > > IMetaStoreClient.showLocks() is deprecated in Hive 2.1 via HIVE-11793. > This method can be removed in Hive 2.2 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13664) test patch
[ https://issues.apache.org/jira/browse/HIVE-13664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13664: -- Attachment: HIVE-13664.2.patch > test patch > -- > > Key: HIVE-13664 > URL: https://issues.apache.org/jira/browse/HIVE-13664 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-13664.2.patch, HIVE-13664.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269936#comment-15269936 ] Eugene Koifman commented on HIVE-13395: --- patch 14 fixes a typo in patch 13 > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.6.patch, > HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T
[jira] [Updated] (HIVE-13395) Lost Update problem in ACID
[ https://issues.apache.org/jira/browse/HIVE-13395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13395: -- Attachment: HIVE-13395.14.patch fix typo > Lost Update problem in ACID > --- > > Key: HIVE-13395 > URL: https://issues.apache.org/jira/browse/HIVE-13395 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.2.0, 2.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13395.11.patch, HIVE-13395.12.patch, > HIVE-13395.13.patch, HIVE-13395.14.patch, HIVE-13395.6.patch, > HIVE-13395.7.patch, HIVE-13395.8.patch > > > ACID users can run into Lost Update problem. > In Hive 1.2, Driver.recordValidTxns() (which records the snapshot to use for > the query) is called in Driver.compile(). > Now suppose to concurrent "update T set x = x + 1" are executed. (for > simplicity assume there is exactly 1 row in T) > What can happen is that both compile at the same time (more precisely before > acquireLocksAndOpenTxn() in runInternal() is called) and thus will lock in > the same snapshot, say the value of x = 7 in this snapshot. > Now 1 will get the lock on the row, the second will block. > Now 1, makes x = 8 and commits. > Now 2 proceeds and makes x = 8 again since in it's snapshot x is still 7. > This specific issue is solved in Hive 1.3/2.0 (HIVE-11077 which is a large > patch that deals with multi-statement txns) by moving recordValidTxns() after > locks are acquired which reduces the likelihood of this but doesn't eliminate > the problem. > > Even in 1.3 version of the code, you could have the same issue. Assume the > same 2 queries: > Both start a txn, say txnid 9 and 10. Say 10 gets the lock first, 9 blocks. > 10 updates the row (so x = 8) and thus ReaderKey.currentTransactionId=10. > 10 commits. > Now 9 can proceed and it will get a snapshot that includes 10, i.e. it will > see x = 8 and it will write x = 9, but it will set > ReaderKey.currentTransactionId = 9. Thus when merge logic runs, it will see > x = 8 is the later version of this row, i.e. lost update. > The problem is that locks alone are insufficient for MVCC architecture. > > At lower level Row ID has (originalTransactionId, rowid, bucket id, > currentTransactionId) and since on update/delete we do a table scan, we could > check that we are about to write a row with currentTransactionId < > (currentTransactionId of row we've read) and fail the query. Currently, > currentTransactionId is not surfaced at higher level where this check can be > made. > This would not work (efficiently) longer term where we want to support fast > update on user defined PK vis streaming ingest. > Also, this would not work with multi statement txns since in that case we'd > lock in the snapshot at the start of the txn, but then 2nd, 3rd etc queries > would use the same snapshot and the locks for these queries would be acquired > after the snapshot is locked in so this would be the same situation as pre > HIVE-11077. > > > A more robust solution (commonly used with MVCC) is to keep track of start > and commit time (logical counter) or each transaction to detect if two txns > overlap. The 2nd part is to keep track of write-set, i.e. which data (rows, > partitions, whatever appropriate level of granularity is) were modified by > any txn and if 2 txns overlap in time and wrote the same element, abort later > one. This is called first-committer-wins rule. This requires a MS DB schema > change > It would be most convenient to use the same sequence for txnId, start and > commit time (in which case txnid=start time). In this case we'd need to add > 1 filed to TXNS table. The complication here is that we'll be using elements > of the sequence faster and they are used as part of file name of delta and > base dir and currently limited to 7 digits which can be exceeded. So this > would require some thought to handling upgrade/migration. > Also, write-set tracking requires either additional metastore table or > keeping info in HIVE_LOCKS around longer with new state. > > In the short term, on SQL side of things we could (in auto commit mode only) > acquire the locks first and then open the txn AND update these locks with txn > id. > This implies another Thrift change to pass in lockId to openTxn. > The same would not work for Streaming API since it opens several txns at once > and then acquires locks for each. > (Not sure if that's is an issue or not since Streaming only does Insert). > Either way this feels hacky. > > Here is one simple example why we need Write-Set tracking for multi-statement > txns > Consider transactions T ~1~ and T ~2~: > T ~1~: r ~1~\[x] -> w ~1~\[y] -> c ~1~ > T ~2~: w ~2~\[x] -> w ~2~\[y]
[jira] [Updated] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12996:
-
Attachment: HIVE-12996.1.patch
patch 1 for test
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12996.1.patch
>
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12996:
-
Status: Patch Available (was: Open)
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12996.1.patch
>
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269920#comment-15269920
]
Wei Zheng commented on HIVE-12996:
--
[~ekoifman] Can you take a look?
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-12996.1.patch
>
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13637 started by Jesus Camacho Rodriguez. -- > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Status: Patch Available (was: In Progress) > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Status: Open (was: Patch Available) > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13637) Fold CASE into NVL when CBO optimized the plan
[ https://issues.apache.org/jira/browse/HIVE-13637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13637: --- Attachment: HIVE-13637.02.patch > Fold CASE into NVL when CBO optimized the plan > -- > > Key: HIVE-13637 > URL: https://issues.apache.org/jira/browse/HIVE-13637 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13637.02.patch, HIVE-13637.patch, HIVE-13637.patch > > > After HIVE-13068 goes in, folding CASE into NVL got disabled when CBO has > optimized the plan, as it was done by ConstantPropagate in Hive. We need to > enable it back. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13646) make hive.optimize.sort.dynamic.partition compatible with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13646: -- Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) committed to branch-1 and master > make hive.optimize.sort.dynamic.partition compatible with ACID tables > - > > Key: HIVE-13646 > URL: https://issues.apache.org/jira/browse/HIVE-13646 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13646.2.patch, HIVE-13646.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries. > dynamic inserts are common in ACID and this leaves users with few options if > they are seeing OutOfMemory errors due to too many writers. > hive.optimize.sort.dynamic.partition sorts data by partition col/bucket > col/sort col to ensure each reducer only needs 1 writer. > Acid requires data in each bucket file to be sorted by ROW__ID and thus > doesn't allow end user to determine sorting. > So we should be able to support hive.optimize.sort.dynamic.partition with > sort on partition col/bucket col/ROW__ID -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13159) TxnHandler should support datanucleus.connectionPoolingType = None
[ https://issues.apache.org/jira/browse/HIVE-13159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269894#comment-15269894 ] Eugene Koifman commented on HIVE-13159: --- I think it would be safer if the un-implemented methods were to throw UnsupportedOperationException or some other exception. Otherwise looks fine. > TxnHandler should support datanucleus.connectionPoolingType = None > -- > > Key: HIVE-13159 > URL: https://issues.apache.org/jira/browse/HIVE-13159 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Sergey Shelukhin >Assignee: Alan Gates > Attachments: HIVE-13159.patch > > > Right now, one has to choose bonecp or dbcp. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13681) Update README with latest Hive functionality
[
https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269889#comment-15269889
]
Lefty Leverenz commented on HIVE-13681:
---
Typo: "analyasis" -> "analysis"
Also, for consistency I recommend using all lowercase for "User Defined
Functions" here:
{code}
+extended with user code via User Defined Functions (UDFs), user defined
+aggregates (UDAFs), and user defined table functions (UDTFs).
{code}
> Update README with latest Hive functionality
>
>
> Key: HIVE-13681
> URL: https://issues.apache.org/jira/browse/HIVE-13681
> Project: Hive
> Issue Type: Improvement
> Components: Documentation
>Affects Versions: 2.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
> Attachments: HIVE-13681.patch
>
>
> In line with the latest changes to the website (see HIVE-13607) and the wiki
> to remove references to Hive QL/HQL and update use cases Hive is appropriate
> for we need to update the README file as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13646) make hive.optimize.sort.dynamic.partition compatible with ACID tables
[ https://issues.apache.org/jira/browse/HIVE-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269842#comment-15269842 ] Eugene Koifman commented on HIVE-13646: --- The fix makes Acid queries respect hive.optimize.sort.dynamic.partition setting. In case of update/delete the query plan is not optimal (it has 2 ReduceSinks which will be addressed in a separate ticket) but now the user has a choice of whether to use hive.optimize.sort.dynamic.partition. thanks Wei for the review > make hive.optimize.sort.dynamic.partition compatible with ACID tables > - > > Key: HIVE-13646 > URL: https://issues.apache.org/jira/browse/HIVE-13646 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-13646.2.patch, HIVE-13646.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries. > dynamic inserts are common in ACID and this leaves users with few options if > they are seeing OutOfMemory errors due to too many writers. > hive.optimize.sort.dynamic.partition sorts data by partition col/bucket > col/sort col to ensure each reducer only needs 1 writer. > Acid requires data in each bucket file to be sorted by ROW__ID and thus > doesn't allow end user to determine sorting. > So we should be able to support hive.optimize.sort.dynamic.partition with > sort on partition col/bucket col/ROW__ID -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269840#comment-15269840
]
Wei Zheng commented on HIVE-12996:
--
The original symptom was that temp table records show up in metastore tables
for ACID. But the root cause is we locked temporary table the same way as we do
for regular table.
Since temp tables are only visible to current session, there cannot be
concurrent operations on them, thus unnecessary to lock them.
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Liew reassigned HIVE-13680: - Assignee: Kevin Liew > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta >Assignee: Kevin Liew > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12996) Temp tables shouldn't be locked
[
https://issues.apache.org/jira/browse/HIVE-12996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Zheng updated HIVE-12996:
-
Summary: Temp tables shouldn't be locked (was: Temp tables shouldn't be
stored in metastore tables for ACID)
> Temp tables shouldn't be locked
> ---
>
> Key: HIVE-12996
> URL: https://issues.apache.org/jira/browse/HIVE-12996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.0.0
>Reporter: Wei Zheng
>Assignee: Wei Zheng
>
> Internally, INSERT INTO ... VALUES statements use temp table to accomplish
> its functionality. But temp tables shouldn't be stored in the metastore
> tables for ACID, because they are by definition only visible inside the
> session that created them, and we don't allow multiple threads inside a
> session. If a temp table is used in a query, it should be ignored by lock
> manager.
> {code}
> mysql> select * from COMPLETED_TXN_COMPONENTS;
> +---+--+---+--+
> | CTC_TXNID | CTC_DATABASE | CTC_TABLE | CTC_PARTITION|
> +---+--+---+--+
> | 1 | acid | t1| NULL |
> | 1 | acid | values__tmp__table__1 | NULL |
> | 2 | acid | t1| NULL |
> | 2 | acid | values__tmp__table__2 | NULL |
> | 3 | acid | values__tmp__table__3 | NULL |
> | 3 | acid | t1| NULL |
> | 4 | acid | values__tmp__table__1 | NULL |
> | 4 | acid | t2p | ds=today |
> | 5 | acid | values__tmp__table__1 | NULL |
> | 5 | acid | t3p | ds=today/hour=12 |
> +---+--+---+--+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269831#comment-15269831 ] Vikram Dixit K commented on HIVE-13620: --- +1. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > HIVE-13620.6.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13646) make hive.optimize.sort.dynamic.partition compatible with ACID tables
[
https://issues.apache.org/jira/browse/HIVE-13646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269820#comment-15269820
]
Eugene Koifman commented on HIVE-13646:
---
{noformat}
Test Name Duration Age
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby8_map
1 min 5 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin_negative3
1.4 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketmapjoin11
0.95 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_cbo_udf_udaf
1 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_skewjoinopt13
1.2 sec 1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_join0 0.86
sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_avro_joins_native
0.74 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_stats130.53
sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby10 0.89
sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_vector_cast_constant
0.83 sec1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_14
0.9 sec 1
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
5.1 sec 2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 6.2 sec 3
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
1 min 23 sec3
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
1 min 16 sec3
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
2.7 sec 6
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
3 sec 7
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
3 sec 7
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
10 sec 8
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
2 min 2 sec 9
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate13 sec
13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
10 sec 13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 35 sec
13
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
20 sec 13
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 49 sec13
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
1.8 sec 13
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.38 sec13
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
3.4 sec 13
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 1 min 32 sec13
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
7.2 sec 13
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
54 sec 13
{noformat}
test failures are not related
> make hive.optimize.sort.dynamic.partition compatible with ACID tables
> -
>
> Key: HIVE-13646
> URL: https://issues.apache.org/jira/browse/HIVE-13646
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-13646.2.patch, HIVE-13646.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries.
> dynamic inserts are common in ACID and this leaves users with few options if
> they are seeing OutOfMemory errors due to too many writers.
> hive.optimize.sort.dynamic.partition sorts data by partition col/bucket
> col/sort col to ensure each reducer only needs 1 writer.
> Acid requires data in each bucket file to be sorted by ROW__ID and thus
> doesn't allow end user to determine sorting.
> So we should be able to support hive.optimize.sort.dynamic.partition with
> sort on partition col/bucket col/ROW__ID
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13682) EOFException with fast hashtable
[
https://issues.apache.org/jira/browse/HIVE-13682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13682:
Description:
While testing something else on recent master, w/Tez 0.8.3, this happened
(TPCDS q27)
{noformat}
Caused by: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:399)
... 20 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
... 4 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.EOFException
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:106)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:304)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:185)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:181)
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:104)
... 5 more
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
at
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readCheckNull(BinarySortableDeserializeRead.java:182)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.putRow(VectorMapJoinFastLongHashTable.java:83)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:181)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:98)
... 9 more
{noformat}
There's no error if fast hashtable is disabled.
was:
While testing something else on recent master, w/Tez 0.8.3, this happened
(TPCDS q27)
{noformat}
Caused by: java.util.concurrent.ExecutionException:
org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hive.ql.exec.Operator.completeInitialization(Operator.java:399)
... 20 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:106)
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache$1.call(LlapObjectCache.java:131)
... 4 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.io.EOFException
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:106)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:304)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:185)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:181)
at
org.apache.hadoop.hive.ql.exec.tez.LlapObjectCache.retrieve(LlapObjectCache.java:104)
... 5 more
Caused by: java.io.EOFException
at
org.apache.hadoop.hive.serde2.binarysortable.InputByteBuffer.read(InputByteBuffer.java:54)
at
org.apache.hadoop.hive.serde2.binarysortable.fast.BinarySortableDeserializeRead.readCheckNull(BinarySortableDeserializeRead.java:182)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastLongHashTable.putRow(VectorMapJoinFastLongHashTable.java:83)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.putRow(VectorMapJoinFastTableContainer.java:181)
at
org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:98)
... 9 more
{noformat}
> EOFException with fast hashtable
>
>
> Key: HIVE-13682
> URL: https://issues.apache.org/jira/browse/HIVE-13682
> Project: Hive
> Issue Type: Bug
>
[jira] [Updated] (HIVE-13653) improve config error messages for LLAP cache size/etc
[ https://issues.apache.org/jira/browse/HIVE-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13653: Attachment: HIVE-13653.01.patch Updated the patch > improve config error messages for LLAP cache size/etc > - > > Key: HIVE-13653 > URL: https://issues.apache.org/jira/browse/HIVE-13653 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13653.01.patch, HIVE-13653.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13653) improve config error messages for LLAP cache size/etc
[ https://issues.apache.org/jira/browse/HIVE-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269802#comment-15269802 ] Sergey Shelukhin commented on HIVE-13653: - The tez tests pass for me, the other appear to be already broken or caused by metastore issues. I will update the patch > improve config error messages for LLAP cache size/etc > - > > Key: HIVE-13653 > URL: https://issues.apache.org/jira/browse/HIVE-13653 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13653.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13119) java.io.IOException: java.io.IOException: ORC does not support type conversion from CHAR to VARCHAR (due to Schema Evolution HIVE-11981)
[
https://issues.apache.org/jira/browse/HIVE-13119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-13119:
Resolution: Cannot Reproduce
Status: Resolved (was: Patch Available)
> java.io.IOException: java.io.IOException: ORC does not support type
> conversion from CHAR to VARCHAR (due to Schema Evolution HIVE-11981)
>
>
> Key: HIVE-13119
> URL: https://issues.apache.org/jira/browse/HIVE-13119
> Project: Hive
> Issue Type: Bug
> Components: Hive, ORC
>Affects Versions: 2.1.0
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-13119.01.patch
>
>
> From e-mail to u...@hive.apache.org
> {noformat}
> New version Hive 2
> a simple Job fails with
>
>
> 2016-02-22T18:23:29,008 ERROR [main]: mr.MapredLocalTask
> (MapredLocalTask.java:executeInProcess(381)) - Hive Runtime Error: Map local
> work failed
> java.io.IOException: java.io.IOException: ORC does not support type
> conversion from CHAR to VARCHAR
>
> This was working OK in Hive 1.2.1! I am using MR here
> --
> Dr Mich Talebzadeh
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HIVE-13683) Remove erroneously included patch file
[ https://issues.apache.org/jira/browse/HIVE-13683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates resolved HIVE-13683. --- Resolution: Fixed File removed. > Remove erroneously included patch file > -- > > Key: HIVE-13683 > URL: https://issues.apache.org/jira/browse/HIVE-13683 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.1.0 > > Attachments: HIVE-13683.patch > > > The commit for HIVE-13509 accidentally included a patch file in the source > tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13683) Remove erroneously included patch file
[ https://issues.apache.org/jira/browse/HIVE-13683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13683: -- Attachment: HIVE-13683.patch > Remove erroneously included patch file > -- > > Key: HIVE-13683 > URL: https://issues.apache.org/jira/browse/HIVE-13683 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Alan Gates >Assignee: Alan Gates > Fix For: 2.1.0 > > Attachments: HIVE-13683.patch > > > The commit for HIVE-13509 accidentally included a patch file in the source > tree. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13264) JDBC driver makes 2 Open Session Calls for every open session
[
https://issues.apache.org/jira/browse/HIVE-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269788#comment-15269788
]
Vaibhav Gumashta edited comment on HIVE-13264 at 5/3/16 10:54 PM:
--
[~nithinmahesh] I thought the issue is with the extra OpenSession RPC call
within HiveConnection#createHttpTransport
(https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L257).
I don't see your patch addressing that.
was (Author: vgumashta):
[~nithinmahesh] I thought the issue with the extra OpenSession RPC call within
HiveConnection#createHttpTransport
(https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L257).
I don't see your patch addressing that.
> JDBC driver makes 2 Open Session Calls for every open session
> -
>
> Key: HIVE-13264
> URL: https://issues.apache.org/jira/browse/HIVE-13264
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: NITHIN MAHESH
>Assignee: NITHIN MAHESH
> Labels: jdbc
> Attachments: HIVE-13264.1.patch, HIVE-13264.2.patch,
> HIVE-13264.3.patch, HIVE-13264.4.patch, HIVE-13264.patch
>
>
> When HTTP is used as the transport mode by the Hive JDBC driver, we noticed
> that there is an additional open/close session just to validate the
> connection.
>
> TCLIService.Iface client = new TCLIService.Client(new
> TBinaryProtocol(transport));
> TOpenSessionResp openResp = client.OpenSession(new TOpenSessionReq());
> if (openResp != null) {
> client.CloseSession(new
> TCloseSessionReq(openResp.getSessionHandle()));
> }
>
> The open session call is a costly one and should not be used to test
> transport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13264) JDBC driver makes 2 Open Session Calls for every open session
[
https://issues.apache.org/jira/browse/HIVE-13264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269788#comment-15269788
]
Vaibhav Gumashta commented on HIVE-13264:
-
[~nithinmahesh] I thought the issue with the extra OpenSession RPC call within
HiveConnection#createHttpTransport
(https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java#L257).
I don't see your patch addressing that.
> JDBC driver makes 2 Open Session Calls for every open session
> -
>
> Key: HIVE-13264
> URL: https://issues.apache.org/jira/browse/HIVE-13264
> Project: Hive
> Issue Type: Bug
> Components: JDBC
>Reporter: NITHIN MAHESH
>Assignee: NITHIN MAHESH
> Labels: jdbc
> Attachments: HIVE-13264.1.patch, HIVE-13264.2.patch,
> HIVE-13264.3.patch, HIVE-13264.4.patch, HIVE-13264.patch
>
>
> When HTTP is used as the transport mode by the Hive JDBC driver, we noticed
> that there is an additional open/close session just to validate the
> connection.
>
> TCLIService.Iface client = new TCLIService.Client(new
> TBinaryProtocol(transport));
> TOpenSessionResp openResp = client.OpenSession(new TOpenSessionReq());
> if (openResp != null) {
> client.CloseSession(new
> TCloseSessionReq(openResp.getSessionHandle()));
> }
>
> The open session call is a costly one and should not be used to test
> transport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13653) improve config error messages for LLAP cache size/etc
[
https://issues.apache.org/jira/browse/HIVE-13653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269764#comment-15269764
]
Hive QA commented on HIVE-13653:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801514/HIVE-13653.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10049 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mapreduce2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_diff_fs
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_5
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.service.cli.session.TestHiveSessionImpl.testLeakOperationHandle
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/164/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/164/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-164/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12801514 - PreCommit-HIVE-MASTER-Build
> improve config error messages for LLAP cache size/etc
> -
>
> Key: HIVE-13653
> URL: https://issues.apache.org/jira/browse/HIVE-13653
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13653.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13516) Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL
[ https://issues.apache.org/jira/browse/HIVE-13516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269760#comment-15269760 ] Alan Gates commented on HIVE-13516: --- +1 > Adding BTEQ .IF, .QUIT, ERRORCODE to HPL/SQL > > > Key: HIVE-13516 > URL: https://issues.apache.org/jira/browse/HIVE-13516 > Project: Hive > Issue Type: Improvement > Components: hpl/sql >Affects Versions: 2.1.0 >Reporter: Dmitry Tolpeko >Assignee: Dmitry Tolpeko > Attachments: HIVE-13516.1.patch > > > Adding Teradata BTEQ features to HPL/SQL such as .IF, .QUIT, ERRORCODE. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to master after rebase, and testing on a cluster. Thanks for the review! > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13442.01.patch, HIVE-13442.nogen.patch, > HIVE-13442.patch, HIVE-13442.patch, HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11165) Calcite planner might have a thread-safety issue compiling in parallel
[ https://issues.apache.org/jira/browse/HIVE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez resolved HIVE-11165. Resolution: Cannot Reproduce I explored this one, and we could not reproduce it the problem. Agreed with [~gopalv] that if it showed up again, we would re-open the issue. > Calcite planner might have a thread-safety issue compiling in parallel > -- > > Key: HIVE-11165 > URL: https://issues.apache.org/jira/browse/HIVE-11165 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez > Attachments: RunJar-2015-06-30.snapshot > > > After about 6 minutes trying to plan a query, the HiveServer2 was killed to > restore functionality to a test run. > The HEP planner is stuck on a TopologicalOrder traversal and there were no > queries being fed into the HiveServer2 after it got stuck. > TPC-DS query13 was the query in question, at 4 way parallel, which triggered > the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11165) Calcite planner might have a thread-safety issue compiling in parallel
[ https://issues.apache.org/jira/browse/HIVE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269751#comment-15269751 ] Julian Hyde commented on HIVE-11165: I don't have an update. It's not obviously a thread-safety issue; the graph which is blowing up in that call stack is not shared between threads. More likely, the planner is firing rules over and over again until the graph of RelNodes gets really large. Thread-safety is one of several possible causes of that. > Calcite planner might have a thread-safety issue compiling in parallel > -- > > Key: HIVE-11165 > URL: https://issues.apache.org/jira/browse/HIVE-11165 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez > Attachments: RunJar-2015-06-30.snapshot > > > After about 6 minutes trying to plan a query, the HiveServer2 was killed to > restore functionality to a test run. > The HEP planner is stuck on a TopologicalOrder traversal and there were no > queries being fed into the HiveServer2 after it got stuck. > TPC-DS query13 was the query in question, at 4 way parallel, which triggered > the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13681) Update README with latest Hive functionality
[ https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13681: -- Status: Patch Available (was: Open) > Update README with latest Hive functionality > > > Key: HIVE-13681 > URL: https://issues.apache.org/jira/browse/HIVE-13681 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13681.patch > > > In line with the latest changes to the website (see HIVE-13607) and the wiki > to remove references to Hive QL/HQL and update use cases Hive is appropriate > for we need to update the README file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13681) Update README with latest Hive functionality
[ https://issues.apache.org/jira/browse/HIVE-13681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13681: -- Attachment: HIVE-13681.patch NO PRECOMMIT TESTS > Update README with latest Hive functionality > > > Key: HIVE-13681 > URL: https://issues.apache.org/jira/browse/HIVE-13681 > Project: Hive > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.0.0 >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13681.patch > > > In line with the latest changes to the website (see HIVE-13607) and the wiki > to remove references to Hive QL/HQL and update use cases Hive is appropriate > for we need to update the README file as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13213) make DbLockManger work for non-acid resources
[ https://issues.apache.org/jira/browse/HIVE-13213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13213: -- Resolution: Fixed Fix Version/s: 2.1.0 1.3.0 Status: Resolved (was: Patch Available) committed to branch-1 and master thanks Alan for the review > make DbLockManger work for non-acid resources > - > > Key: HIVE-13213 > URL: https://issues.apache.org/jira/browse/HIVE-13213 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 1.3.0, 2.1.0 > > Attachments: HIVE-13213.patch > > > for example, > insert into T values(...) > if T is an ACID table we acquire Read lock > but for non-acid table it should acquire Exclusive lock -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11165) Calcite planner might have a thread-safety issue compiling in parallel
[ https://issues.apache.org/jira/browse/HIVE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269736#comment-15269736 ] Sergey Shelukhin commented on HIVE-11165: - Any update here? [~jcamachorodriguez] [~julianhyde] > Calcite planner might have a thread-safety issue compiling in parallel > -- > > Key: HIVE-11165 > URL: https://issues.apache.org/jira/browse/HIVE-11165 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.0.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez > Attachments: RunJar-2015-06-30.snapshot > > > After about 6 minutes trying to plan a query, the HiveServer2 was killed to > restore functionality to a test run. > The HEP planner is stuck on a TopologicalOrder traversal and there were no > queries being fed into the HiveServer2 after it got stuck. > TPC-DS query13 was the query in question, at 4 way parallel, which triggered > the issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11848) tables in subqueries don't get locked
[
https://issues.apache.org/jira/browse/HIVE-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-11848:
--
Resolution: Fixed
Fix Version/s: 2.1.0
1.3.0
Status: Resolved (was: Patch Available)
committed to branch-1 and master
thanks Wei for the review
> tables in subqueries don't get locked
> -
>
> Key: HIVE-11848
> URL: https://issues.apache.org/jira/browse/HIVE-11848
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Fix For: 1.3.0, 2.1.0
>
> Attachments: HIVE-11848.patch
>
>
> Consider
> {noformat}
> update acidTbl set b=19 where acidTbl.b in(select I.b from nonAcidOrcTbl I
> where I.a = 3)
> {noformat}
> noAcidOrcTbl doesn't get locked at all. (SHARED_WRITE is taken on acidTbl).
> Same for __delete__ with subquery
> This is is because the ReadEntity for nonAcidOrcTbl is skipped by
> {noformat}
> for (ReadEntity input : plan.getInputs()) {
> if (!input.needsLock() || input.isUpdateOrDelete()) {
> // We don't want to acquire readlocks during update or delete as
> we'll be acquiring write
> // locks instead.
> continue;
> }
> {noformat}
> whatever sets isUpdateOrDelete() flag doesn't pay attention to whether the
> table is written to or not.
> HIVE-10150 was a similar issue, abstractly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13607) Change website references to HQL/HiveQL to SQL
[ https://issues.apache.org/jira/browse/HIVE-13607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-13607: -- Resolution: Fixed Status: Resolved (was: Patch Available) Patch committed to svn and website published. Thanks [~leftylev] for the review. > Change website references to HQL/HiveQL to SQL > -- > > Key: HIVE-13607 > URL: https://issues.apache.org/jira/browse/HIVE-13607 > Project: Hive > Issue Type: Improvement > Components: Website >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: HIVE-13607.2.patch, HIVE-13607.patch > > > When it started Hive's SQL dialect was far enough from standard SQL that the > developers called it HQL or HiveQL. > Over the years Hive's SQL dialect has matured. It still has some oddities > but it is explicitly pushing towards SQL 2011 conformance. Calling the > language anything but SQL now is confusing for users. > In addition to changing the website I propose to make changes in the wiki. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13657) Spark driver stderr logs should appear in hive client logs
[
https://issues.apache.org/jira/browse/HIVE-13657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269727#comment-15269727
]
Szehon Ho commented on HIVE-13657:
--
+1. Seems very useful, i havent tried myself but was wondering does the stderr
also contain a lot of non-relevant things, and any way to not include those?
> Spark driver stderr logs should appear in hive client logs
> --
>
> Key: HIVE-13657
> URL: https://issues.apache.org/jira/browse/HIVE-13657
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-13657.patch
>
>
> Currently, spark driver exceptions are not getting logged in beeline.
> Instead, the users sees the not-so-useful:
> {code}
> ERROR : Failed to execute spark task, with exception
> 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark
> client.)'
>
> {code}
> The user has to look at HS2 logs to discover the root cause:
> {code}
> 2015-04-01 11:33:16,048 INFO org.apache.hive.spark.client.SparkClientImpl:
> 15/04/01 11:33:16 WARN UserGroupInformation: PriviledgedActionException
> as:foo (auth:PROXY) via hive (auth:SIMPLE)
> cause:org.apache.hadoop.security.AccessControlException: Permission denied:
> user=foo, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
> ...
> {code}
> We should surface these critical errors in hive client.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13659) An empty where condition leads to vectorization exceptions instead of throwing a compile time error
[
https://issues.apache.org/jira/browse/HIVE-13659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269723#comment-15269723
]
Ashutosh Chauhan commented on HIVE-13659:
-
Patch LGTM. But I think reported failures of TestVectorFilterOperator &
TestVectorizer are related.
> An empty where condition leads to vectorization exceptions instead of
> throwing a compile time error
> ---
>
> Key: HIVE-13659
> URL: https://issues.apache.org/jira/browse/HIVE-13659
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Fix For: 2.1.0
>
> Attachments: HIVE-13659.01.patch
>
>
> A partial query
> select count (distinct field) from table where field;
> Note the missing 'field=value'
> resulted in the following error in task logs, instead of failing early during
> compile
> {code}
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:91)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:68)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:326)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:150)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:52)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
> ... 17 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector cannot be cast to
> org.apache.hadoop.hive.ql.exec.vector.LongColumnVector
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.SelectColumnIsTrue.evaluate(SelectColumnIsTrue.java:46)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:106)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:45)
> ... 18 more
> {code}
> Comment by Gunther:
> I think this works by implicitly converting the expr to boolean (if a cast is
> necessary). This query looks legal to me and probably needs to be handle in
> Vectorization
> Comment by Ashutosh:
> Oracle, postgres & sql server throws error for this if type of field is not
> boolean. However, MySQL & Hive (with vectorization off) executes the query by
> implicitly adding a cast to boolean. Hive shall be consistent in its behavior
> regardless whether vectorization is on or off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13679) Pass diagnostic message to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13679: --- Status: Patch Available (was: Open) > Pass diagnostic message to failure hooks > > > Key: HIVE-13679 > URL: https://issues.apache.org/jira/browse/HIVE-13679 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13679.1.patch > > > Pass diagnostic message to failure hooks. This is useful for debugging remote > job failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13679) Pass diagnostic message to failure hooks
[ https://issues.apache.org/jira/browse/HIVE-13679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HIVE-13679: --- Attachment: HIVE-13679.1.patch > Pass diagnostic message to failure hooks > > > Key: HIVE-13679 > URL: https://issues.apache.org/jira/browse/HIVE-13679 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Attachments: HIVE-13679.1.patch > > > Pass diagnostic message to failure hooks. This is useful for debugging remote > job failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13442: Hadoop Flags: Incompatible change > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.01.patch, HIVE-13442.nogen.patch, > HIVE-13442.patch, HIVE-13442.patch, HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13680: Description: With HIVE-12049 in, we can provide an option to compress ResultSets before writing to disk. The user can specify a compression library via a config param which can be used in the tasks. (was: We can provide an option to compress ResultSets before writing to disk) > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta > > With HIVE-12049 in, we can provide an option to compress ResultSets before > writing to disk. The user can specify a compression library via a config > param which can be used in the tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13680: Description: We can provide an option to compress ResultSets before writing to disk > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta > > We can provide an option to compress ResultSets before writing to disk -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10438) HiveServer2: Enable Type specific ResultSet compression
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269702#comment-15269702 ] Vaibhav Gumashta commented on HIVE-10438: - Created HIVE-13680 to compress ResultSet using a configurable plugin for all data types. That can be worked on before this jira. > HiveServer2: Enable Type specific ResultSet compression > --- > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Kevin Liew > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13680: Component/s: JDBC HiveServer2 > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, JDBC >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13660) Vectorizing IN expression with list of columns throws java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to ExprNodeConstantDesc
[ https://issues.apache.org/jira/browse/HIVE-13660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269695#comment-15269695 ] Prasanth Jayachandran commented on HIVE-13660: -- +1 > Vectorizing IN expression with list of columns throws > java.lang.ClassCastException ExprNodeColumnDesc cannot be cast to > ExprNodeConstantDesc > > > Key: HIVE-13660 > URL: https://issues.apache.org/jira/browse/HIVE-13660 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-13660.01.patch > > > Example: > SELECT * FROM alltypesorc WHERE cint in (ctinyint, cbigint); -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-10438) HiveServer2: Enable Type specific ResultSet compression
[ https://issues.apache.org/jira/browse/HIVE-10438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-10438: Summary: HiveServer2: Enable Type specific ResultSet compression (was: HiveServer2: Enable ResultSet compression for HS2/*DBC driver) > HiveServer2: Enable Type specific ResultSet compression > --- > > Key: HIVE-10438 > URL: https://issues.apache.org/jira/browse/HIVE-10438 > Project: Hive > Issue Type: Sub-task > Components: Hive, Thrift API >Affects Versions: 1.2.0 >Reporter: Rohit Dholakia >Assignee: Kevin Liew > Labels: patch > Attachments: HIVE-10438-1.patch, HIVE-10438.patch, > Proposal-rscompressor.pdf, README.txt, > Results_Snappy_protobuf_TBinary_TCompact.pdf, hs2ResultSetCompressor.zip, > hs2driver-master.zip > > > This JIRA proposes an architecture for enabling ResultSet compression which > uses an external plugin. > The patch has three aspects to it: > 0. An architecture for enabling ResultSet compression with external plugins > 1. An example plugin to demonstrate end-to-end functionality > 2. A container to allow everyone to write and test ResultSet compressors with > a query submitter (https://github.com/xiaom/hs2driver) > Also attaching a design document explaining the changes, experimental results > document, and a pdf explaining how to setup the docker container to observe > end-to-end functionality of ResultSet compression. > https://reviews.apache.org/r/35792/ Review board link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13680) HiveServer2: Provide a way to compress ResultSets
[ https://issues.apache.org/jira/browse/HIVE-13680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-13680: Issue Type: Sub-task (was: Improvement) Parent: HIVE-12427 > HiveServer2: Provide a way to compress ResultSets > - > > Key: HIVE-13680 > URL: https://issues.apache.org/jira/browse/HIVE-13680 > Project: Hive > Issue Type: Sub-task >Reporter: Vaibhav Gumashta > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[ https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13638: --- Status: Patch Available (was: In Progress) > CBO rule to pull up constants through Sort/Limit > > > Key: HIVE-13638 > URL: https://issues.apache.org/jira/browse/HIVE-13638 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13638.01.patch, HIVE-13638.patch > > > After HIVE-13068 goes in, we need to pull up constants through Sort/Limit > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[ https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13638: --- Attachment: HIVE-13638.01.patch > CBO rule to pull up constants through Sort/Limit > > > Key: HIVE-13638 > URL: https://issues.apache.org/jira/browse/HIVE-13638 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13638.01.patch, HIVE-13638.patch > > > After HIVE-13068 goes in, we need to pull up constants through Sort/Limit > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[ https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-13638 started by Jesus Camacho Rodriguez. -- > CBO rule to pull up constants through Sort/Limit > > > Key: HIVE-13638 > URL: https://issues.apache.org/jira/browse/HIVE-13638 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13638.patch > > > After HIVE-13068 goes in, we need to pull up constants through Sort/Limit > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13638) CBO rule to pull up constants through Sort/Limit
[ https://issues.apache.org/jira/browse/HIVE-13638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-13638: --- Status: Open (was: Patch Available) > CBO rule to pull up constants through Sort/Limit > > > Key: HIVE-13638 > URL: https://issues.apache.org/jira/browse/HIVE-13638 > Project: Hive > Issue Type: Sub-task > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-13638.patch > > > After HIVE-13068 goes in, we need to pull up constants through Sort/Limit > operator, as it was done previously by ConstantPropagate in Hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13178) Enhance ORC Schema Evolution to handle more standard data type conversions
[ https://issues.apache.org/jira/browse/HIVE-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13178: Attachment: HIVE-13178.094.patch Actual patch checked in -- adjusted for Q file outputs. > Enhance ORC Schema Evolution to handle more standard data type conversions > -- > > Key: HIVE-13178 > URL: https://issues.apache.org/jira/browse/HIVE-13178 > Project: Hive > Issue Type: Bug > Components: Hive, ORC >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 2.1.0 > > Attachments: HIVE-13178.01.patch, HIVE-13178.02.patch, > HIVE-13178.03.patch, HIVE-13178.04.patch, HIVE-13178.05.patch, > HIVE-13178.06.patch, HIVE-13178.07.patch, HIVE-13178.08.patch, > HIVE-13178.09.patch, HIVE-13178.091.patch, HIVE-13178.092.patch, > HIVE-13178.093.patch, HIVE-13178.094.patch > > > Currently, SHORT -> INT -> BIGINT is supported. > Handle ORC data type conversions permitted by Implicit conversion allowed by > TypeIntoUtils.implicitConvertible method. >* STRING_GROUP -> DOUBLE >* STRING_GROUP -> DECIMAL >* DATE_GROUP -> STRING >* NUMERIC_GROUP -> STRING >* STRING_GROUP -> STRING_GROUP >* >* // Upward from "lower" type to "higher" numeric type: >* BYTE -> SHORT -> INT -> BIGINT -> FLOAT -> DOUBLE -> DECIMAL -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-11793) SHOW LOCKS with DbTxnManager ignores filter options
[ https://issues.apache.org/jira/browse/HIVE-11793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-11793: - Attachment: HIVE-11793.4.patch upload patch 4, fixed a problem of showing tables of the same name from different databases [~ekoifman] Can you take another look? > SHOW LOCKS with DbTxnManager ignores filter options > --- > > Key: HIVE-11793 > URL: https://issues.apache.org/jira/browse/HIVE-11793 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng >Priority: Minor > Attachments: HIVE-11793.1.patch, HIVE-11793.2.patch, > HIVE-11793.3.patch, HIVE-11793.4.patch > > > https://cwiki.apache.org/confluence/display/Hive/Locking and > https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-ShowLocks > list various options that can be used with SHOW LOCKS, e.g. > When ACID is enabled, all these options are ignored and a full list is > returned. > (also only ext lock id is shown, int lock id is not). > see DDLTask.showLocks() and TxnHandler.showLocks() > requires extending ShowLocksRequest which is a Thrift object -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13442) LLAP: refactor submit API to be amenable to signing
[ https://issues.apache.org/jira/browse/HIVE-13442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269598#comment-15269598 ] Sergey Shelukhin commented on HIVE-13442: - What HS2 provides is a separate patch. It will provide the LLAP token and the fragment stuff; what other credentials are necessary? In case of Tez, I assume it will just work as it does already. > LLAP: refactor submit API to be amenable to signing > --- > > Key: HIVE-13442 > URL: https://issues.apache.org/jira/browse/HIVE-13442 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13442.01.patch, HIVE-13442.nogen.patch, > HIVE-13442.patch, HIVE-13442.patch, HIVE-13442.protobuf.patch > > > This is going to be a wire compat breaking change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[ https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13391: Attachment: HIVE-13391.08.patch Rebase > add an option to LLAP to use keytab to authenticate to read data > > > Key: HIVE-13391 > URL: https://issues.apache.org/jira/browse/HIVE-13391 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13391.01.patch, HIVE-13391.02.patch, > HIVE-13391.03.patch, HIVE-13391.04.patch, HIVE-13391.05.patch, > HIVE-13391.06.patch, HIVE-13391.07.patch, HIVE-13391.08.patch, > HIVE-13391.patch > > > This can be used for non-doAs case to allow access to clients who don't > propagate HDFS tokens. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13391) add an option to LLAP to use keytab to authenticate to read data
[
https://issues.apache.org/jira/browse/HIVE-13391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269569#comment-15269569
]
Hive QA commented on HIVE-13391:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12801833/HIVE-13391.07.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 9968 tests
executed
*Failed tests:*
{noformat}
TestHBaseAggrStatsCacheIntegration - did not produce a TEST-*.xml file
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-union2.q-bucket4.q-and-12-more - did
not produce a TEST-*.xml file
TestMiniTezCliDriver-vectorized_parquet.q-orc_merge4.q-orc_merge_incompat1.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.llap.daemon.impl.comparator.TestShortestJobFirstComparator.testWaitQueueComparatorWithinDagPriority
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote.org.apache.hadoop.hive.metastore.TestAuthzApiEmbedAuthorizerInRemote
org.apache.hadoop.hive.metastore.TestFilterHooks.org.apache.hadoop.hive.metastore.TestFilterHooks
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty
org.apache.hadoop.hive.metastore.TestMetaStoreEndFunctionListener.testEndFunctionListener
org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus
org.apache.hadoop.hive.metastore.TestMetaStoreInitListener.testMetaStoreInitListener
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.org.apache.hadoop.hive.metastore.TestMetaStoreMetrics
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAddPartitionWithValidPartVal
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithCommas
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithUnicode
org.apache.hadoop.hive.metastore.TestPartitionNameWhitelistValidation.testAppendPartitionWithValidCharacters
org.apache.hadoop.hive.metastore.TestRetryingHMSHandler.testRetryingHMSHandler
org.apache.hadoop.hive.ql.security.TestClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestExtendedAcls.org.apache.hadoop.hive.ql.security.TestExtendedAcls
org.apache.hadoop.hive.ql.security.TestFolderPermissions.org.apache.hadoop.hive.ql.security.TestFolderPermissions
org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener.org.apache.hadoop.hive.ql.security.TestMultiAuthorizationPreEventListener
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testDelegationTokenSharedStore
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testSaslWithHiveMetaStore
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
org.apache.hive.hcatalog.listener.TestDbNotificationListener.cleanupNotifs
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.org.apache.hive.minikdc.TestJdbcWithDBTokenStore
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/163/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/163/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-163/
Messages:
{noformat}
[jira] [Updated] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13351: - Attachment: (was: HIVE-13351.2.patch) > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13351: - Attachment: HIVE-13351.2.patch > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Patch Available (was: Open) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Status: Open (was: Patch Available) > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13445) LLAP: token should encode application and cluster ids
[ https://issues.apache.org/jira/browse/HIVE-13445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-13445: Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) committed to master > LLAP: token should encode application and cluster ids > - > > Key: HIVE-13445 > URL: https://issues.apache.org/jira/browse/HIVE-13445 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 2.1.0 > > Attachments: HIVE-13445.01.patch, HIVE-13445.02.patch, > HIVE-13445.03.patch, HIVE-13445.04.patch, HIVE-13445.05.patch, > HIVE-13445.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13341) Stats state is not captured correctly: differentiate load table and create table
[ https://issues.apache.org/jira/browse/HIVE-13341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13341: --- Attachment: HIVE-13341.08.patch address create table stats. > Stats state is not captured correctly: differentiate load table and create > table > > > Key: HIVE-13341 > URL: https://issues.apache.org/jira/browse/HIVE-13341 > Project: Hive > Issue Type: Sub-task > Components: Logical Optimizer, Statistics >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13341.01.patch, HIVE-13341.02.patch, > HIVE-13341.03.patch, HIVE-13341.04.patch, HIVE-13341.05.patch, > HIVE-13341.06.patch, HIVE-13341.07.patch, HIVE-13341.08.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-13620: -- Attachment: HIVE-13620.6.patch patch v6 with newest merge from master plus [~vikram.dixit]'s review feedback. > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > HIVE-13620.6.patch, llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13598) Describe extended table should show the primary keys/foreign keys associated with the table
[ https://issues.apache.org/jira/browse/HIVE-13598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269528#comment-15269528 ] Ashutosh Chauhan commented on HIVE-13598: - We also need to add support for show create table statement. If its involved, we can take it up as a follow-up. > Describe extended table should show the primary keys/foreign keys associated > with the table > --- > > Key: HIVE-13598 > URL: https://issues.apache.org/jira/browse/HIVE-13598 > Project: Hive > Issue Type: Bug >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13598.1.patch, HIVE-13598.2.patch, > HIVE-13598.3.patch, HIVE-13598.4.patch > > > After HIVE-13290 is committed, we need to show the constraints as part of > table description when extended label is used. Currently, the constraints > would not be shown as part of table description since Constraint is a > separate entity. > The purpose of the jira is to modify Hive.describeTable() to enable the user > to view the constraints associated with the table when the user does a > "describe extended table"; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11848) tables in subqueries don't get locked
[
https://issues.apache.org/jira/browse/HIVE-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269526#comment-15269526
]
Eugene Koifman commented on HIVE-11848:
---
reran all failed tests with age=1 All passed
> tables in subqueries don't get locked
> -
>
> Key: HIVE-11848
> URL: https://issues.apache.org/jira/browse/HIVE-11848
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-11848.patch
>
>
> Consider
> {noformat}
> update acidTbl set b=19 where acidTbl.b in(select I.b from nonAcidOrcTbl I
> where I.a = 3)
> {noformat}
> noAcidOrcTbl doesn't get locked at all. (SHARED_WRITE is taken on acidTbl).
> Same for __delete__ with subquery
> This is is because the ReadEntity for nonAcidOrcTbl is skipped by
> {noformat}
> for (ReadEntity input : plan.getInputs()) {
> if (!input.needsLock() || input.isUpdateOrDelete()) {
> // We don't want to acquire readlocks during update or delete as
> we'll be acquiring write
> // locks instead.
> continue;
> }
> {noformat}
> whatever sets isUpdateOrDelete() flag doesn't pay attention to whether the
> table is written to or not.
> HIVE-10150 was a similar issue, abstractly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269522#comment-15269522 ] Ashutosh Chauhan commented on HIVE-13351: - +1 pending tests > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9174) Enable queuing of HCatalog notification events in metastore DB
[ https://issues.apache.org/jira/browse/HIVE-9174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269478#comment-15269478 ] Sravya Tirukkovalur commented on HIVE-9174: --- Curious, which jira added the sql files corresponding to this change? > Enable queuing of HCatalog notification events in metastore DB > -- > > Key: HIVE-9174 > URL: https://issues.apache.org/jira/browse/HIVE-9174 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: TODOC15 > Fix For: 1.1.0 > > Attachments: HIVE-9174.patch > > > HCatalog supports notifying listeners of events such as table creation, etc. > using the MetaStoreEventListener interface. The only non-test implementation > of this interface today is NotificationListerner, which currently sends > notifications via JMS. This requires a JMS service to be set up in the > cluster. > This JIRA proposes adding a new DbNotificationListener, that will store > events directly in the metastore, and allow users to access those events via > HiveMetastoreClient. This avoids the need for a 3rd tool to install and > manage when listening for events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13620) Merge llap branch work to master
[ https://issues.apache.org/jira/browse/HIVE-13620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269471#comment-15269471 ] Gunther Hagleitner commented on HIVE-13620: --- Patch looks good to me. +1. (I think v5 isn't on rb yet.) > Merge llap branch work to master > > > Key: HIVE-13620 > URL: https://issues.apache.org/jira/browse/HIVE-13620 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-13620.1.patch, HIVE-13620.2.patch, > HIVE-13620.3.patch, HIVE-13620.4.patch, HIVE-13620.5.patch, > llap_master_diff.txt > > > Would like to try to merge the llap branch work for HIVE-12991 into the > master branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11848) tables in subqueries don't get locked
[
https://issues.apache.org/jira/browse/HIVE-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269467#comment-15269467
]
Eugene Koifman commented on HIVE-11848:
---
{noformat}
Test Name Duration Age
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join32 7.3 sec 1
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
1 min 31 sec1
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
1 min 19 sec1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
10 sec 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
42 ms 1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_short_regress
6 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_aggregate_9
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_vec_mapwork_part
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_part
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_diff_fs
2 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_2
2 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_14
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_orc_nonvec_mapwork_table
1 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_multi_insert
0 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce1
4 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_nested_mapjoin
0 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_order_null
3 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_groupby_reduce
0 ms1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
2 ms1
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskCommunicator.testFinishableStateUpdateFailure
5.1 sec 4
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testPreemptionQueueComparator
5 sec 4
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testFetchingPartitionsWithDifferentSchemas
3 sec 5
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testAddPartitions
3 sec 5
org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault
10 sec 6
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
1 min 42 sec7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_selectindate21 sec
11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avrocountemptytbl
10 sec 11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_null 35 sec
11
org.apache.hive.hcatalog.api.repl.commands.TestCommands.org.apache.hive.hcatalog.api.repl.commands.TestCommands
20 sec 11
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
1 min 32 sec11
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForMemoryTokenStore
2 sec 11
org.apache.hive.minikdc.TestHiveAuthFactory.testStartTokenManagerForDBTokenStore
0.35 sec11
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
3.3 sec 11
org.apache.hive.minikdc.TestMiniHiveKdc.testLogin 2 min 0 sec 11
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
8.4 sec 11
org.apache.hadoop.hive.cli.TestMinimrCliDriver.org.apache.hadoop.hive.cli.TestMinimrCliDriver
50 sec 11
{noformat}
> tables in subqueries don't get locked
> -
>
> Key: HIVE-11848
> URL: https://issues.apache.org/jira/browse/HIVE-11848
> Project: Hive
> Issue Type: Bug
> Components: Locking, Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-11848.patch
>
>
> Consider
> {noformat}
> update acidTbl set b=19 where acidTbl.b in(select I.b from nonAcidOrcTbl I
> where I.a = 3)
> {noformat}
> noAcidOrcTbl doesn't get locked at all. (SHARED_WRITE is taken on acidTbl).
> Same for __delete__ with subquery
> This is is because the ReadEntity for
[jira] [Updated] (HIVE-4924) JDBC: Support query timeout for jdbc
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-4924:
---
Fix Version/s: 2.1.0
> JDBC: Support query timeout for jdbc
>
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Navis
>Assignee: Vaibhav Gumashta
> Fix For: 2.1.0
>
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.4.patch,
> HIVE-4924.5.patch, HIVE-4924.6.patch, HIVE-4924.7.patch,
> HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-4924) JDBC: Support query timeout for jdbc
[
https://issues.apache.org/jira/browse/HIVE-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-4924:
---
Target Version/s: 2.1.0
> JDBC: Support query timeout for jdbc
>
>
> Key: HIVE-4924
> URL: https://issues.apache.org/jira/browse/HIVE-4924
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, JDBC
>Affects Versions: 2.0.0
>Reporter: Navis
>Assignee: Vaibhav Gumashta
> Fix For: 2.1.0
>
> Attachments: HIVE-4924.3.patch.txt, HIVE-4924.4.patch,
> HIVE-4924.5.patch, HIVE-4924.6.patch, HIVE-4924.7.patch,
> HIVE-4924.D11787.1.patch, HIVE-4924.D11787.2.patch
>
>
> support
> {code}
> Statement.setQueryTimeout(int timeout)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13490) Change itests to be part of the main Hive build
[
https://issues.apache.org/jira/browse/HIVE-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269430#comment-15269430
]
Zoltan Haindrich commented on HIVE-13490:
-
I've also attached the itests to my top level project; and i've played a lot
with eclipse to be able to execute most of the unit tests directly from eclipse
; plus the qtests...I'm glad that it works now ;)
But it's not very easy to set it up...i've uncovered a few things - which could
be addressed programatically, or by using some conventions:
* there are a few path related properties (${test.data.files} and friends)
which could be defaulted to some path if the project root is known - the
project root can be detected by a few simple checks (for example existence of
the {{ql}} directory...or anything special) - I would definetly keep the finer
grained properties - but they wouldn't be mandatory anymore
* every test execution gets it's hive configuration xmls thru an extra
classpath entry supplied by the maven surefire plugin - if the configuration
could be kept in a unique directory - it could be loaded more precisely without
problems for the ide.
* i'm not a maven expert but it might be possible to mask the itests / test
task by using some kind of profile - this might prevent {{mvn test}} from
recklessly executing integration tests by default
are there any intrest in these?
> Change itests to be part of the main Hive build
> ---
>
> Key: HIVE-13490
> URL: https://issues.apache.org/jira/browse/HIVE-13490
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-13490.01.patch, HIVE-13490.02.patch
>
>
> Instead of having to build Hive, and then itests separately.
> With IntelliJ, this ends up being loaded as two separate dependencies, and
> there's a lot of hops involved to make changes.
> Does anyone know why these have been kept separate ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269395#comment-15269395
]
Sergio Peña commented on HIVE-13632:
Thanks [~ychena]. The patch looks good.
+1
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch, HIVE-13632.2.patch,
> HIVE-13632.3.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13351: - Status: Open (was: Patch Available) > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13351) Support drop Primary Key/Foreign Key constraints
[ https://issues.apache.org/jira/browse/HIVE-13351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Sankar Sivarama Subramaniyan updated HIVE-13351: - Attachment: HIVE-13351.2.patch > Support drop Primary Key/Foreign Key constraints > > > Key: HIVE-13351 > URL: https://issues.apache.org/jira/browse/HIVE-13351 > Project: Hive > Issue Type: Sub-task > Components: CBO, Logical Optimizer >Reporter: Hari Sankar Sivarama Subramaniyan >Assignee: Hari Sankar Sivarama Subramaniyan > Attachments: HIVE-13351.1.patch, HIVE-13351.2.patch > > > ALTER TABLE TABLENAME DROP CONSTRAINT CONSTRAINTNAME; > The CONSTRAINTNAME has to be associated with the mentioned table, i.e. there > should be atleast 1 table column of TABLENAME participating in the constraint. > Otherwise, we should throw an error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13507) Improved logging for ptest
[ https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13507: --- Assignee: Siddharth Seth (was: Sergio Peña) > Improved logging for ptest > -- > > Key: HIVE-13507 > URL: https://issues.apache.org/jira/browse/HIVE-13507 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.1.0 > > Attachments: HIVE-13507.01.patch, HIVE-13507.02.patch > > > NO PRECOMMIT TESTS > Include information about batch runtimes, outlier lists, host completion > times, etc. Try identifying tests which cause the build to take a long time > while holding onto resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13507) Improved logging for ptest
[ https://issues.apache.org/jira/browse/HIVE-13507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-13507: --- Description: NO PRECOMMIT TESTS Include information about batch runtimes, outlier lists, host completion times, etc. Try identifying tests which cause the build to take a long time while holding onto resources. was:Include information about batch runtimes, outlier lists, host completion times, etc. Try identifying tests which cause the build to take a long time while holding onto resources. > Improved logging for ptest > -- > > Key: HIVE-13507 > URL: https://issues.apache.org/jira/browse/HIVE-13507 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth >Assignee: Sergio Peña > Fix For: 2.1.0 > > Attachments: HIVE-13507.01.patch, HIVE-13507.02.patch > > > NO PRECOMMIT TESTS > Include information about batch runtimes, outlier lists, host completion > times, etc. Try identifying tests which cause the build to take a long time > while holding onto resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13027) Configuration changes to improve logging performance
[ https://issues.apache.org/jira/browse/HIVE-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269349#comment-15269349 ] Sergey Shelukhin commented on HIVE-13027: - +1 assuming it actually works and logs to correct files :) > Configuration changes to improve logging performance > > > Key: HIVE-13027 > URL: https://issues.apache.org/jira/browse/HIVE-13027 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Attachments: HIVE-13027.1.patch, HIVE-13027.2.patch, > HIVE-13027.3.patch > > > For LLAP and HS2, some configuration changes can be made to improve logging > performance > 1) LOG4j2's async logger claims to have 6-68 times better performance than > synchronous logger. https://logging.apache.org/log4j/2.x/manual/async.html > 2) Replace File appenders with RandomAccessFileAppender that claims to be > 20-200% more performant. > https://logging.apache.org/log4j/2.x/manual/appenders.html#RandomAccessFileAppender > Also make async logging configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13674) usingTezAm field not required in LLAP SubmitWorkRequestProto
[ https://issues.apache.org/jira/browse/HIVE-13674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15269307#comment-15269307 ] Sergey Shelukhin commented on HIVE-13674: - It should be good, I was just suggesting for future reference > usingTezAm field not required in LLAP SubmitWorkRequestProto > > > Key: HIVE-13674 > URL: https://issues.apache.org/jira/browse/HIVE-13674 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Fix For: llap > > Attachments: HIVE-13674.1.patch > > > From [~sseth] during review of HIVE-13620, the usingTezAm field is not needed > in SubmitWorkRequestProto. > Also, SendEventsRequestProto/SendEventsResponseProto are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13632) Hive failing on insert empty array into parquet table
[
https://issues.apache.org/jira/browse/HIVE-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongzhi Chen updated HIVE-13632:
Attachment: HIVE-13632.3.patch
> Hive failing on insert empty array into parquet table
> -
>
> Key: HIVE-13632
> URL: https://issues.apache.org/jira/browse/HIVE-13632
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Affects Versions: 1.1.0
>Reporter: Yongzhi Chen
>Assignee: Yongzhi Chen
> Attachments: HIVE-13632.1.patch, HIVE-13632.2.patch,
> HIVE-13632.3.patch
>
>
> The insert will fail with following stack:
> {noformat}
> by: parquet.io.ParquetEncodingException: empty fields are illegal, the field
> should be ommited completely instead
> at
> parquet.io.MessageColumnIO$MessageColumnIORecordConsumer.endField(MessageColumnIO.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$ListDataWriter.write(DataWritableWriter.java:271)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$GroupDataWriter.write(DataWritableWriter.java:199)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter$MessageDataWriter.write(DataWritableWriter.java:215)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriter.write(DataWritableWriter.java:88)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:59)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.DataWritableWriteSupport.write(DataWritableWriteSupport.java:31)
> at
> parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:116)
> at
> parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:123)
> at parquet.hadoop.ParquetRecordWriter.write(ParquetRecordWriter.java:42)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:111)
> at
> org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:697)
> {noformat}
> Reproduce:
> {noformat}
> create table test_small (
> key string,
> arrayValues array)
> stored as parquet;
> insert into table test_small select 'abcd', array() from src limit 1;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
