[jira] [Updated] (HIVE-18559) Allow hive.metastore.limit.partition.request to be user configurable
[ https://issues.apache.org/jira/browse/HIVE-18559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-18559: --- Status: Patch Available (was: Open) > Allow hive.metastore.limit.partition.request to be user configurable > > > Key: HIVE-18559 > URL: https://issues.apache.org/jira/browse/HIVE-18559 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Minor > Attachments: HIVE-18559.patch > > > HIVE-13884 introduced hive.metastore.limit.partition.request to limit the > number of partitions that can be requested from the Metastore for a given > table. > This config is set by cluster admins to prevent ad-hoc queries from putting > memory pressure on HMS. But the limit can be too restrictive for some (savvy) > users who require the ability to set a higher limit for a subset of workloads > (during low HMS memory pressure, for example). Such users generally instruct > admins not to set this confg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18559) Allow hive.metastore.limit.partition.request to be user configurable
[ https://issues.apache.org/jira/browse/HIVE-18559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-18559: --- Attachment: HIVE-18559.patch > Allow hive.metastore.limit.partition.request to be user configurable > > > Key: HIVE-18559 > URL: https://issues.apache.org/jira/browse/HIVE-18559 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Minor > Attachments: HIVE-18559.patch > > > HIVE-13884 introduced hive.metastore.limit.partition.request to limit the > number of partitions that can be requested from the Metastore for a given > table. > This config is set by cluster admins to prevent ad-hoc queries from putting > memory pressure on HMS. But the limit can be too restrictive for some (savvy) > users who require the ability to set a higher limit for a subset of workloads > (during low HMS memory pressure, for example). Such users generally instruct > admins not to set this confg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18559) Allow hive.metastore.limit.partition.request to be user configurable
[ https://issues.apache.org/jira/browse/HIVE-18559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal reassigned HIVE-18559: -- > Allow hive.metastore.limit.partition.request to be user configurable > > > Key: HIVE-18559 > URL: https://issues.apache.org/jira/browse/HIVE-18559 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Minor > > HIVE-13884 introduced hive.metastore.limit.partition.request to limit the > number of partitions that can be requested from the Metastore for a given > table. > This config is set by cluster admins to prevent ad-hoc queries from putting > memory pressure on HMS. But the limit can be too restrictive for some (savvy) > users who require the ability to set a higher limit for a subset of workloads > (during low HMS memory pressure, for example). Such users generally instruct > admins not to set this confg. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.2.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.2.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: (was: HIVE-15305.1.patch) > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[
https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113276#comment-16113276
]
Mohit Sabharwal commented on HIVE-15305:
Thanks, [~vgumashta]. Re-attaching the patch. Appreciate any insights you have
on the sqlInsertPartition failure, which is passing for me locally.
{{assertEquals(24, rsp.getEventsSize());}} is failing with rsp.getEventsSize()
returning 17.
> Add tests for METASTORE_EVENT_LISTENERS
> ---
>
> Key: HIVE-15305
> URL: https://issues.apache.org/jira/browse/HIVE-15305
> Project: Hive
> Issue Type: Bug
>Reporter: Mohit Sabharwal
>Assignee: Mohit Sabharwal
> Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch,
> HIVE-15305.1.patch, HIVE-15305.patch
>
>
> HIVE-15232 reused TestDbNotificationListener to test
> METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of
> METASTORE_EVENT_LISTENERS config. We should test both.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, > HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16107351#comment-16107351 ] Mohit Sabharwal commented on HIVE-15305: The TestTransactionalDbNotificationListener#sqlInsertPartition failure looks related. But it's not failing for me locally. Re-attaching patch for re-run the tests. > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.1.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.1.patch, HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16106526#comment-16106526 ] Mohit Sabharwal commented on HIVE-15305: Incorporated review feedback and rebased patch. Apologies for the delay on this. [~vgumashta], could you please take a look ? RB link: https://reviews.apache.org/r/61244/ > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17088) HS2 WebUI throws a NullPointerException when opened
[
https://issues.apache.org/jira/browse/HIVE-17088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16100171#comment-16100171
]
Mohit Sabharwal commented on HIVE-17088:
[~spena], could you add some comment/description of why apache-jsp dependency
is needed. Not clear from the NPE. Thanks.
> HS2 WebUI throws a NullPointerException when opened
> ---
>
> Key: HIVE-17088
> URL: https://issues.apache.org/jira/browse/HIVE-17088
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Fix For: 3.0.0
>
> Attachments: HIVE-17088.1.patch, HIVE-17088.addendum1.patch
>
>
> After bumping the Jetty version to 3.9 and excluding several other
> dependencies on HIVE-16049, the HS2 webui stopped working and throwing a NPE
> error.
> {noformat}
> HTTP ERROR 500
> Problem accessing /hiveserver2.jsp. Reason:
> Server Error
> Caused by:
> java.lang.NullPointerException
> at
> org.apache.hive.generated.hiveserver2.hiveserver2_jsp._jspService(hiveserver2_jsp.java:181)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:840)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:240)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Powered by Jetty:// 9.3.19.v20170502
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17117) Metalisteners are not notified when threadlocal metaconf is cleanup
[ https://issues.apache.org/jira/browse/HIVE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16096147#comment-16096147 ] Mohit Sabharwal commented on HIVE-17117: Thanks. LGTM, +1 pending tests. > Metalisteners are not notified when threadlocal metaconf is cleanup > > > Key: HIVE-17117 > URL: https://issues.apache.org/jira/browse/HIVE-17117 > Project: Hive > Issue Type: Bug > Components: Metastore > Environment: Tested on master branch (Applicable for downlevel > versions as well) >Reporter: PRASHANT GOLASH >Assignee: PRASHANT GOLASH >Priority: Minor > Attachments: HIVE-17117.1.patch, HIVE-17117.patch > > > Meta listeners are not notified of meta-conf cleanup. This could potentially > leave stale values on listeners objects. For e.g. > Request1 > a. HS2 -> HMS : HMSHandler#setMetaConf > MetaListeners are notified of the ConfigChangeEvent. > b. HS2 -> HMS : HMSHandler#shutdown / HiveMetaStore#deleteContext (if > shutdown is not invoked) > MetaConf is cleaned up in HiveMetaStore#cleanupRawStore, but meta > listeners are not notified > Request 2 > 3. HS2->HMS : AlterPartition > MetaListeners are notified of AlterPartitionEvent. If any listener has > taken dependency on the meta conf value, it will still be having stale value > from Request1 and would potentially be having issues. > The correct behavior should be to notify meta listeners on cleanup as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17117) Metalisteners are not notified when threadlocal metaconf is cleanup
[ https://issues.apache.org/jira/browse/HIVE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095091#comment-16095091 ] Mohit Sabharwal commented on HIVE-17117: looks like the patch attached needs to be refreshed. Not the same as one on RB. > Metalisteners are not notified when threadlocal metaconf is cleanup > > > Key: HIVE-17117 > URL: https://issues.apache.org/jira/browse/HIVE-17117 > Project: Hive > Issue Type: Bug > Components: Metastore > Environment: Tested on master branch (Applicable for downlevel > versions as well) >Reporter: PRASHANT GOLASH >Assignee: PRASHANT GOLASH >Priority: Minor > Attachments: HIVE-17117.patch > > > Meta listeners are not notified of meta-conf cleanup. This could potentially > leave stale values on listeners objects. For e.g. > Request1 > a. HS2 -> HMS : HMSHandler#setMetaConf > MetaListeners are notified of the ConfigChangeEvent. > b. HS2 -> HMS : HMSHandler#shutdown / HiveMetaStore#deleteContext (if > shutdown is not invoked) > MetaConf is cleaned up in HiveMetaStore#cleanupRawStore, but meta > listeners are not notified > Request 2 > 3. HS2->HMS : AlterPartition > MetaListeners are notified of AlterPartitionEvent. If any listener has > taken dependency on the meta conf value, it will still be having stale value > from Request1 and would potentially be having issues. > The correct behavior should be to notify meta listeners on cleanup as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17117) Metalisteners are not notified when threadlocal metaconf is cleanup
[ https://issues.apache.org/jira/browse/HIVE-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16092510#comment-16092510 ] Mohit Sabharwal commented on HIVE-17117: Thanks [~pgolash], left a few comments on RB. > Metalisteners are not notified when threadlocal metaconf is cleanup > > > Key: HIVE-17117 > URL: https://issues.apache.org/jira/browse/HIVE-17117 > Project: Hive > Issue Type: Bug > Components: Metastore > Environment: Tested on master branch (Applicable for downlevel > versions as well) >Reporter: PRASHANT GOLASH >Assignee: PRASHANT GOLASH >Priority: Minor > Attachments: HIVE-17117.patch > > > Meta listeners are not notified of meta-conf cleanup. This could potentially > leave stale values on listeners objects. For e.g. > Request1 > a. HS2 -> HMS : HMSHandler#setMetaConf > MetaListeners are notified of the ConfigChangeEvent. > b. HS2 -> HMS : HMSHandler#shutdown / HiveMetaStore#deleteContext (if > shutdown is not invoked) > MetaConf is cleaned up in HiveMetaStore#cleanupRawStore, but meta > listeners are not notified > Request 2 > 3. HS2->HMS : AlterPartition > MetaListeners are notified of AlterPartitionEvent. If any listener has > taken dependency on the meta conf value, it will still be having stale value > from Request1 and would potentially be having issues. > The correct behavior should be to notify meta listeners on cleanup as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077493#comment-16077493
]
Mohit Sabharwal edited comment on HIVE-17048 at 7/7/17 2:22 PM:
+1, LGTM
was (Author: mohitsabharwal):
1, LGTM
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077493#comment-16077493
]
Mohit Sabharwal commented on HIVE-17048:
1, LGTM
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch, HIVE-17048.2.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-17022: --- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks Naveen Gangam for the review. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17022.1.patch, HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076630#comment-16076630
]
Mohit Sabharwal commented on HIVE-17008:
{{ListenerEvent}} has the success flag, so listener implementation can choose
to ignore failed events. But {{DbNotificationListener}} and
{{HMSMetricsListener}} are ignoring the status flag. The old
{{NotificationListener}} though is looking at the flag and only logging
successful events.
This does look like something we should make consistent, not just address the
NPE. [~sushanth], [~thejas], [~akolb], do you have any thoughts ?
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17048) Pass HiveOperation info to HiveSemanticAnalyzerHook through HiveSemanticAnalyzerHookContext
[
https://issues.apache.org/jira/browse/HIVE-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075769#comment-16075769
]
Mohit Sabharwal commented on HIVE-17048:
Good to add the operation type to TestHs2Hooks and TestHs2HooksWithMiniKdc unit
tests as well (see HIVE-8338)
LGTM, otherwise.
> Pass HiveOperation info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext
> ---
>
> Key: HIVE-17048
> URL: https://issues.apache.org/jira/browse/HIVE-17048
> Project: Hive
> Issue Type: Improvement
> Components: Hooks
>Affects Versions: 2.1.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-17048.1.patch
>
>
> Currently hive passes the following info to HiveSemanticAnalyzerHook through
> HiveSemanticAnalyzerHookContext (see
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L553).
> But the operation type (HiveOperation) is also needed in some cases, e.g.,
> when integrating with Sentry.
> {noformat}
> hookCtx.setConf(conf);
> hookCtx.setUserName(userName);
> hookCtx.setIpAddress(SessionState.get().getUserIpAddress());
> hookCtx.setCommand(command);
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075763#comment-16075763
]
Mohit Sabharwal commented on HIVE-17008:
Thanks, [~dan_impala_9180]. There are two flavors of listeners in each DDL
operation, one which runs in the same transaction as the DDL event (notify only
upon success) and one which run outside the transaction (can notify failed DDL
operations as well). Where exactly is the NPE - I assume somewhere down the
notifyEvent stack ?
+ [~spena], who has worked on this recently.
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075566#comment-16075566 ] Mohit Sabharwal commented on HIVE-17022: Test failures are unrelated. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.1.patch, HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-17022: --- Attachment: HIVE-17022.1.patch > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.1.patch, HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075199#comment-16075199 ] Mohit Sabharwal commented on HIVE-17022: Thanks. Note that Logger is from slf4j not log4j, so there is no extra cost of string formatting. But its probably better to add the conditional for readability. The other lock is really just a helper, so making it private. Printed sorted locks is a great idea. Updating patch. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.1.patch, HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075030#comment-16075030 ] Mohit Sabharwal commented on HIVE-17022: For a table will large number of partitions, it will indeed print lot of log statements. But only applicable under debug mode, when you are debugging and could use this info. For a complex query involving many write entities, it is hard to tell what lock is being taken for what entity otherwise. Earlier, we had these statements as INFO, which we changed to DEBUG in HIVE-12966 to avoid noise. We already have the debug statement for ZooKeeperHiveLockManager, but did not print the actual lock mode in that statement, so I added that to this patch. For EmbeddedLockManager, which is useful when debugging locally, we had no debug statements whatsoever, so I added that to this patch. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17008) HiveMetastore.drop_database can return NPE if database does not exist
[
https://issues.apache.org/jira/browse/HIVE-17008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074983#comment-16074983
]
Mohit Sabharwal commented on HIVE-17008:
[~dan_impala_9180], the changes aren't clear to me, look like changes in
indentation. Could you add a review board link ?
Also, could you add the NPE stracktrace you are seeing to the description ?
You also need a unit test here, perhaps in TestDbNotificationListener
> HiveMetastore.drop_database can return NPE if database does not exist
> -
>
> Key: HIVE-17008
> URL: https://issues.apache.org/jira/browse/HIVE-17008
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Reporter: Dan Burkert
>Assignee: Dan Burkert
> Attachments: HIVE-17008.0.patch
>
>
> When dropping a non-existent database, the HMS will still fire registered
> {{DROP_DATABASE}} event listeners. This results in an NPE when the listeners
> attempt to deref the {{null}} database parameter.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074828#comment-16074828 ] Mohit Sabharwal commented on HIVE-17022: Test failures are unrelated. > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-17022: --- Status: Patch Available (was: Open) > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-17022: --- Attachment: HIVE-17022.patch > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > Attachments: HIVE-17022.patch > > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17022) Add mode in lock debug statements
[ https://issues.apache.org/jira/browse/HIVE-17022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal reassigned HIVE-17022: -- > Add mode in lock debug statements > - > > Key: HIVE-17022 > URL: https://issues.apache.org/jira/browse/HIVE-17022 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal >Priority: Trivial > > Currently, lock debug statements print IMPLICIT/EXPLICIT as lock mode, > whereas SHARED/EXCLUSIVE/SEMI_SHARED are more useful > when debugging. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16738) Notification ID generation in DBNotification might not be unique across HS2 instances.
[
https://issues.apache.org/jira/browse/HIVE-16738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021190#comment-16021190
]
Mohit Sabharwal commented on HIVE-16738:
[~anishek], not sure I understand. Note that ObjectStore/RawStore is a thread
local (see HiveMetaStore.java).
> Notification ID generation in DBNotification might not be unique across HS2
> instances.
> --
>
> Key: HIVE-16738
> URL: https://issues.apache.org/jira/browse/HIVE-16738
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
> Fix For: 3.0.0
>
>
> Going to explain the problem in scope of "replication" feature for hive 2
> that is being built, as it is easier to explain:
> To allow replication to work we need to set
> "hive.metastore.transactional.event.listeners" to DBNotificationListener.
> For use cases where there are multiple HiveServer2 Instances running
> {code}
> private void process(NotificationEvent event, ListenerEvent listenerEvent)
> throws MetaException {
> event.setMessageFormat(msgFactory.getMessageFormat());
> synchronized (NOTIFICATION_TBL_LOCK) {
> LOG.debug("DbNotificationListener: Processing : {}:{}",
> event.getEventId(),
> event.getMessage());
> HMSHandler.getMSForConf(hiveConf).addNotificationEvent(event);
> }
> // Set the DB_NOTIFICATION_EVENT_ID for future reference by other
> listeners.
> if (event.isSetEventId()) {
> listenerEvent.putParameter(
> MetaStoreEventListenerConstants.DB_NOTIFICATION_EVENT_ID_KEY_NAME,
> Long.toString(event.getEventId()));
> }
> }
> {code}
> the above code in DBNotificationListner having the object lock wont be
> guarantee enough to make sure that all events get a unique id. The
> transaction isolation level at the db "read-comitted" or "repeatable-read"
> would also not guarantee the same, unless a lock is at the db level
> preferably on table {{NOTIFICATION_SEQUENCE}} which only has one row.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16164) Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.
[
https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15954302#comment-15954302
]
Mohit Sabharwal commented on HIVE-16164:
+1, LGTM pending tests
> Provide mechanism for passing HMS notification ID between transactional and
> non-transactional listeners.
>
>
> Key: HIVE-16164
> URL: https://issues.apache.org/jira/browse/HIVE-16164
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch,
> HIVE-16164.3.patch, HIVE-16164.6.patch, HIVE-16164.7.patch, HIVE-16164.8.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS
> backend DB so that external applications (such as backup apps) can request
> incremental notifications based on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However,
> there are sometimes that applications may be required to be in sync with the
> latest HMS event in order to process an action. These applications will
> provide a listener implementation that is called by the HMS after an HMS
> transaction happened.
> The problem is that the listener running after the transaction (or during the
> non-transactional context) may need the DB event ID in order to sync all
> events happened previous to that event ID, but this ID is never passed to the
> non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on
> each ListenerEvent implementations (such as CreateTableEvent), and send the
> EnvironmentContext to the non-transactional listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the
> ObjectStore.addNotificationEvent(). We just need to set this event ID to the
> EnvironmentContext from each of the event notifications and make sure that
> this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
> ms.createTable(tbl);
> if (transactionalListeners.size() > 0) {
> CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
> createTableEvent.setEnvironmentContext(envContext);
> for (MetaStoreEventListener transactionalListener :
> transactionalListeners) {
> transactionalListener.onCreateTable(createTableEvent); // <-
> Here the notification ID is generated
> }
> }
> success = ms.commitTransaction();
> } finally {
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(tblPath, true);
> }
> }
> for (MetaStoreEventListener listener : listeners) {
> CreateTableEvent createTableEvent =
> new CreateTableEvent(tbl, success, this);
> createTableEvent.setEnvironmentContext(envContext);
> listener.onCreateTable(createTableEvent);// <-
> Here we would like to consume notification ID
> }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext,
> such as DB_NOTIFICATION_EVENT_ID.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager
[ https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15934893#comment-15934893 ] Mohit Sabharwal commented on HIVE-15766: Thanks, [~vgumashta], latest patch LGTM. Sorry about the late response. > DBNotificationlistener leaks JDOPersistenceManager > -- > > Key: HIVE-15766 > URL: https://issues.apache.org/jira/browse/HIVE-15766 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.0.0, 2.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Fix For: 2.2.0 > > Attachments: HIVE-15766.1.patch, HIVE-15766.2.patch, > HIVE-15766.3.patch, HIVE-15766.4.patch, HIVE-15766.5.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16164) Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.
[
https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926901#comment-15926901
]
Mohit Sabharwal commented on HIVE-16164:
One listener is called only on success, the other can get called even upon
failure.
The listener implementation can decide whether failed event should be handled
or ignored. So, I think we can safely re-use the event among the two listeners.
> Provide mechanism for passing HMS notification ID between transactional and
> non-transactional listeners.
>
>
> Key: HIVE-16164
> URL: https://issues.apache.org/jira/browse/HIVE-16164
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS
> backend DB so that external applications (such as backup apps) can request
> incremental notifications based on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However,
> there are sometimes that applications may be required to be in sync with the
> latest HMS event in order to process an action. These applications will
> provide a listener implementation that is called by the HMS after an HMS
> transaction happened.
> The problem is that the listener running after the transaction (or during the
> non-transactional context) may need the DB event ID in order to sync all
> events happened previous to that event ID, but this ID is never passed to the
> non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on
> each ListenerEvent implementations (such as CreateTableEvent), and send the
> EnvironmentContext to the non-transactional listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the
> ObjectStore.addNotificationEvent(). We just need to set this event ID to the
> EnvironmentContext from each of the event notifications and make sure that
> this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
> ms.createTable(tbl);
> if (transactionalListeners.size() > 0) {
> CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
> createTableEvent.setEnvironmentContext(envContext);
> for (MetaStoreEventListener transactionalListener :
> transactionalListeners) {
> transactionalListener.onCreateTable(createTableEvent); // <-
> Here the notification ID is generated
> }
> }
> success = ms.commitTransaction();
> } finally {
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(tblPath, true);
> }
> }
> for (MetaStoreEventListener listener : listeners) {
> CreateTableEvent createTableEvent =
> new CreateTableEvent(tbl, success, this);
> createTableEvent.setEnvironmentContext(envContext);
> listener.onCreateTable(createTableEvent);// <-
> Here we would like to consume notification ID
> }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext,
> such as DB_NOTIFICATION_EVENT_ID.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16164) Provide mechanism for passing HMS notification ID between transactional and non-transactional listeners.
[
https://issues.apache.org/jira/browse/HIVE-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926850#comment-15926850
]
Mohit Sabharwal commented on HIVE-16164:
I think using EnvironmentContext for writing internal state so that it
can be passed around seems unclean to me. The motivation behind
EnvironmentContext was to allow users to specify arbitrary key value
pairs, like partition names for add_partitions call (HIVE-2994). IOW,
the properties are user provided, arbitrary in nature and so
should be considered strictly read-only.
Why don't we simply add an id field in ListnerEvent ? This field can be
null for any event that does not have a id associated with it (i.e.
for events that are not persisted in db). This will also do away
with unnecessarily creating a whole thrift EnvironmentContext
object when no property was passed by the user.
The MetaStoreListenerNotifier looks like a nice cleanup.
> Provide mechanism for passing HMS notification ID between transactional and
> non-transactional listeners.
>
>
> Key: HIVE-16164
> URL: https://issues.apache.org/jira/browse/HIVE-16164
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-16164.1.patch, HIVE-16164.2.patch
>
>
> The HMS DB notification listener currently stores an event ID on the HMS
> backend DB so that external applications (such as backup apps) can request
> incremental notifications based on the last event ID requested.
> The HMS DB notification and backup applications are asynchronous. However,
> there are sometimes that applications may be required to be in sync with the
> latest HMS event in order to process an action. These applications will
> provide a listener implementation that is called by the HMS after an HMS
> transaction happened.
> The problem is that the listener running after the transaction (or during the
> non-transactional context) may need the DB event ID in order to sync all
> events happened previous to that event ID, but this ID is never passed to the
> non-transactional listeners.
> We can pass this event information through the EnvironmentContext found on
> each ListenerEvent implementations (such as CreateTableEvent), and send the
> EnvironmentContext to the non-transactional listeners to get the event ID.
> The DbNotificactionListener already knows the event ID after calling the
> ObjectStore.addNotificationEvent(). We just need to set this event ID to the
> EnvironmentContext from each of the event notifications and make sure that
> this EnvironmentContext is sent to the non-transactional listeners.
> Here's the code example when creating a table on {{create_table_core}}:
> {noformat}
> ms.createTable(tbl);
> if (transactionalListeners.size() > 0) {
> CreateTableEvent createTableEvent = new CreateTableEvent(tbl, true, this);
> createTableEvent.setEnvironmentContext(envContext);
> for (MetaStoreEventListener transactionalListener :
> transactionalListeners) {
> transactionalListener.onCreateTable(createTableEvent); // <-
> Here the notification ID is generated
> }
> }
> success = ms.commitTransaction();
> } finally {
> if (!success) {
> ms.rollbackTransaction();
> if (madeDir) {
> wh.deleteDir(tblPath, true);
> }
> }
> for (MetaStoreEventListener listener : listeners) {
> CreateTableEvent createTableEvent =
> new CreateTableEvent(tbl, success, this);
> createTableEvent.setEnvironmentContext(envContext);
> listener.onCreateTable(createTableEvent);// <-
> Here we would like to consume notification ID
> }
> {noformat}
> We could use a specific key name that will be used on the EnvironmentContext,
> such as DB_NOTIFICATION_EVENT_ID.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager
[ https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890107#comment-15890107 ] Mohit Sabharwal commented on HIVE-15766: Wondering if it's better to create a new static version of getMS that takes hiveConf as argument. That way, both existing getMS() and DbNotificationHandler can call this method. It will allow us to use the same code path as current getMS() (including ms.verifySchema() call etc.) and newRawStore() can be static as well. Also wondering if there is any point storing rawStoreClassName separately in HMSHandler since it can be looked up via hiveConf and it's only used once when RawStore gets created. > DBNotificationlistener leaks JDOPersistenceManager > -- > > Key: HIVE-15766 > URL: https://issues.apache.org/jira/browse/HIVE-15766 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15766.1.patch, HIVE-15766.2.patch, > HIVE-15766.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15766) DBNotificationlistener leaks JDOPersistenceManager
[ https://issues.apache.org/jira/browse/HIVE-15766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884421#comment-15884421 ] Mohit Sabharwal commented on HIVE-15766: LGTM, +1 A small nit that process() assumes that HMSHandler.getRawStore() threadlocal is initialized. That threadlocal gets initialized inside HMSHandler.getMS() which, in practice always gets called before process() ( because we persist metadata before notification) -- so it will work in practice. Maybe good to put a null check to be safe. > DBNotificationlistener leaks JDOPersistenceManager > -- > > Key: HIVE-15766 > URL: https://issues.apache.org/jira/browse/HIVE-15766 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta > Attachments: HIVE-15766.1.patch, HIVE-15766.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16016) Use same PersistenceManager for metadata and notification
[ https://issues.apache.org/jira/browse/HIVE-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-16016: --- Resolution: Duplicate Status: Resolved (was: Patch Available) Marking this as duplicate of HIVE-15766. [~vgumashta], it'd be great to commit that patch, because it's a critical fix. Fixed the failing tests in this patch and moved those over to HIVE-15305., appreciate if you could take a look at it as well. Thank you! > Use same PersistenceManager for metadata and notification > - > > Key: HIVE-16016 > URL: https://issues.apache.org/jira/browse/HIVE-16016 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-16016.patch > > > HIVE-13966 added to support for persisting notification in the same JDO > transaction as the metadata event. However, the notification is currently > added using a different ObjectStore object from the one used to persist the > metadata event. > The notification is added using the ObjectStore constructed in > DbNotificationListener, whereas the metadata event is added via the thread > local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). > As a result, different PersistenceManagers (different transactions) are used > to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883113#comment-15883113 ] Mohit Sabharwal commented on HIVE-15305: [~vgumashta], could you please a take a look ? Thanks! > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Status: Patch Available (was: Open) > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15305) Add tests for METASTORE_EVENT_LISTENERS
[ https://issues.apache.org/jira/browse/HIVE-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15305: --- Attachment: HIVE-15305.patch > Add tests for METASTORE_EVENT_LISTENERS > --- > > Key: HIVE-15305 > URL: https://issues.apache.org/jira/browse/HIVE-15305 > Project: Hive > Issue Type: Bug >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15305.patch > > > HIVE-15232 reused TestDbNotificationListener to test > METASTORE_TRANSACTIONAL_EVENT_LISTENERS and removed unit testing of > METASTORE_EVENT_LISTENERS config. We should test both. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16016) Use same PersistenceManager for metadata and notification
[ https://issues.apache.org/jira/browse/HIVE-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879753#comment-15879753 ] Mohit Sabharwal commented on HIVE-16016: Thanks for the review [~sershe]! The separate RS in DbNotificationListener is used by the CleanerThread (which gets created by DbNotificationListener) outside the TThreadPoolServer/hmshandler threadpool. Thanks, [~vgumashta]. Looks like your patch already had a successful test run. Please commit your patch. I'll move the test portion of this patch over to HIVE-15305 (where it really belongs). > Use same PersistenceManager for metadata and notification > - > > Key: HIVE-16016 > URL: https://issues.apache.org/jira/browse/HIVE-16016 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-16016.patch > > > HIVE-13966 added to support for persisting notification in the same JDO > transaction as the metadata event. However, the notification is currently > added using a different ObjectStore object from the one used to persist the > metadata event. > The notification is added using the ObjectStore constructed in > DbNotificationListener, whereas the metadata event is added via the thread > local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). > As a result, different PersistenceManagers (different transactions) are used > to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16016) Use same PersistenceManager for metadata and notification
[ https://issues.apache.org/jira/browse/HIVE-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-16016: --- Attachment: HIVE-16016.patch > Use same PersistenceManager for metadata and notification > - > > Key: HIVE-16016 > URL: https://issues.apache.org/jira/browse/HIVE-16016 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-16016.patch > > > HIVE-13966 added to support for persisting notification in the same JDO > transaction as the metadata event. However, the notification is currently > added using a different ObjectStore object from the one used to persist the > metadata event. > The notification is added using the ObjectStore constructed in > DbNotificationListener, whereas the metadata event is added via the thread > local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). > As a result, different PersistenceManagers (different transactions) are used > to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16016) Use same PersistenceManager for metadata and notification
[ https://issues.apache.org/jira/browse/HIVE-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-16016: --- Status: Patch Available (was: Open) > Use same PersistenceManager for metadata and notification > - > > Key: HIVE-16016 > URL: https://issues.apache.org/jira/browse/HIVE-16016 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-16016.patch > > > HIVE-13966 added to support for persisting notification in the same JDO > transaction as the metadata event. However, the notification is currently > added using a different ObjectStore object from the one used to persist the > metadata event. > The notification is added using the ObjectStore constructed in > DbNotificationListener, whereas the metadata event is added via the thread > local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). > As a result, different PersistenceManagers (different transactions) are used > to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16016) Use same PersistenceManager for metadata and notification
[ https://issues.apache.org/jira/browse/HIVE-16016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal reassigned HIVE-16016: -- > Use same PersistenceManager for metadata and notification > - > > Key: HIVE-16016 > URL: https://issues.apache.org/jira/browse/HIVE-16016 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > > HIVE-13966 added to support for persisting notification in the same JDO > transaction as the metadata event. However, the notification is currently > added using a different ObjectStore object from the one used to persist the > metadata event. > The notification is added using the ObjectStore constructed in > DbNotificationListener, whereas the metadata event is added via the thread > local ObjectStore (i.e. threadLocalMS in HiveMetaStore.HMSHandler). > As a result, different PersistenceManagers (different transactions) are used > to persist notification and metadata events. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15710) HS2 Stopped when running in background
[
https://issues.apache.org/jira/browse/HIVE-15710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868655#comment-15868655
]
Mohit Sabharwal commented on HIVE-15710:
[~Ferd], proposal to only do background logic for beeline and hs2 looks good to
me.
> HS2 Stopped when running in background
> --
>
> Key: HIVE-15710
> URL: https://issues.apache.org/jira/browse/HIVE-15710
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Attachments: HIVE-15710.1.patch, HIVE-15710.2.patch
>
>
> To reproduce, start HS2 in background like {{hive --service hiveserver2 &}},
> and run some query from beeline using Tez or Spark as engine.
> I think it's similar to HIVE-6758: HS2 uses jline for the in-place progress
> update, and receives SIGTTOU when trying to call stty.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15361) INSERT dynamic partition on S3 fails with a MoveTask failure
[
https://issues.apache.org/jira/browse/HIVE-15361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729267#comment-15729267
]
Mohit Sabharwal commented on HIVE-15361:
LGTM, +1
> INSERT dynamic partition on S3 fails with a MoveTask failure
>
>
> Key: HIVE-15361
> URL: https://issues.apache.org/jira/browse/HIVE-15361
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>Priority: Critical
> Attachments: HIVE-15361.1.patch, HIVE-15361.2.patch,
> HIVE-15361.3.patch, HIVE-15361.4.patch, HIVE-15361.5.patch
>
>
> The following failure is due to the patch that merges two MoveTask found on
> the ConditionalTask (See HIVE-15114)
> {panel:title=Repro steps}
> CREATE EXTERNAL TABLE external_1k0jU (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING);
> ALTER TABLE external_1k0jU ADD PARTITION (COUNTRY='USA', STATE='CA');
> INSERT INTO external_1k0jU PARTITION (country='USA', state='CA') values
> ('John Doe', 23), ('Jane Doe', 22);
> CREATE EXTERNAL TABLE external_P3kiT (name STRING, age INT) PARTITIONED BY
> (country STRING, state STRING) location 's3a://hive-on-s3/foo/bar/';
> set hive.exec.dynamic.partition.mode=nonstrict;
> INSERT INTO TABLE external_P3kiT PARTITION (country, state) SELECT * FROM
> external_1k0jU;
> {panel}
> {panel:title=Error & stack trace}
> ERROR : FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.MoveTask
> INFO : MapReduce Jobs Launched:
> INFO : Stage-Stage-1: Map: 1 Cumulative CPU: 3.64 sec HDFS Read: 3656
> HDFS Write: 99 SUCCESS
> INFO : Total MapReduce CPU Time Spent: 3 seconds 640 msec
> INFO : Completed executing
> command(queryId=hive_20161201113939_d64df5d7-a4c4-4885-846f-10f0223fcf4c);
> Time taken: 23.227 seconds
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.MoveTask (state=08S01,code=1)
> INFO : Loading data to table default.external_p3kit partition (country=null,
> state=null) from
> s3a://hive-on-s3/foo/bar/.hive-staging_hive_2016-12-01_11-39-48_741_6724911837889341086-13/-ext-10002
> {code}
> ERROR : Failed with exception MetaException(message:Invalid partition key &
> values; keys [country, state, ], values [])
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Invalid partition key & values; keys [country, state,
> ], values [])
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1902)
> at org.apache.hadoop.hive.ql.metadata.Hive.getPartition(Hive.java:1834)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1428)
> at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1388)
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:453)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1976)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1689)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1421)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1205)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796)
> at
> org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: MetaException(message:Invalid partition key & values; keys
> [country, state, ], values [])
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_with_auth_result$get_partition_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java:65142)
> at
>
[jira] [Commented] (HIVE-15363) Execute hive-blobstore tests using ProxyLocalFileSystem
[
https://issues.apache.org/jira/browse/HIVE-15363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725929#comment-15725929
]
Mohit Sabharwal commented on HIVE-15363:
lgtm, +1
> Execute hive-blobstore tests using ProxyLocalFileSystem
> ---
>
> Key: HIVE-15363
> URL: https://issues.apache.org/jira/browse/HIVE-15363
> Project: Hive
> Issue Type: Test
> Components: Hive
>Reporter: Sergio Peña
>Assignee: Sergio Peña
> Attachments: HIVE-15363.1.patch
>
>
> The {{hive-blobstore}} directory contains tests that an only be executed on
> blobstorage systems currently. These test are run manually by committers.
> To automate these tests on HiveQA, we should allow hive-blobstore to use the
> ProxyLocalFileSystem to run more test coverage on the pre-commit jenkins jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15341) Get work path instead of attempted task path in HiveHFileOutputFormat
[ https://issues.apache.org/jira/browse/HIVE-15341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15722692#comment-15722692 ] Mohit Sabharwal commented on HIVE-15341: LGTM, +1 > Get work path instead of attempted task path in HiveHFileOutputFormat > - > > Key: HIVE-15341 > URL: https://issues.apache.org/jira/browse/HIVE-15341 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang >Priority: Minor > Attachments: HIVE-15341.patch > > > It would be more robust to use FileOutputCommitter.getWorkPath instead of > FileOutputCommitter.getTaskAttemptPath. > The getTaskAttemptPath is same as getWorkPath in MR2 new APIs but is missing > in MR1 old APIs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706116#comment-15706116 ] Mohit Sabharwal commented on HIVE-15232: [~vgumashta], was on vacation so couldn't respond earlier. You're right, this patch should not remove unit testing for the older config. Created HIVE-15305 to fix this. Thanks! > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Fix For: 2.2.0 > > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15246) Add a making comment to blobstore staging paths on qtest output
[ https://issues.apache.org/jira/browse/HIVE-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680594#comment-15680594 ] Mohit Sabharwal commented on HIVE-15246: LGTM, +1 > Add a making comment to blobstore staging paths on qtest output > --- > > Key: HIVE-15246 > URL: https://issues.apache.org/jira/browse/HIVE-15246 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-15246.1.patch > > > HIVE-15226 added partial masking comments to qtest output, but hive-staging > directories on blobstore are still full masked. We need to mask these > partially as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13539) HiveHFileOutputFormat searching the wrong directory for HFiles
[
https://issues.apache.org/jira/browse/HIVE-13539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677272#comment-15677272
]
Mohit Sabharwal commented on HIVE-13539:
LGTM, +1
> HiveHFileOutputFormat searching the wrong directory for HFiles
> --
>
> Key: HIVE-13539
> URL: https://issues.apache.org/jira/browse/HIVE-13539
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 1.1.0
> Environment: Built into CDH 5.4.7
>Reporter: Tim Robertson
>Assignee: Chaoyu Tang
>Priority: Blocker
> Attachments: HIVE-13539.1.patch, HIVE-13539.patch,
> hive_hfile_output_format.q, hive_hfile_output_format.q.out
>
>
> When creating HFiles for a bulkload in HBase I believe it is looking in the
> wrong directory to find the HFiles, resulting in the following exception:
> {code}
> Error: java.lang.RuntimeException: Hive Runtime Error while closing
> operators: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:295)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:188)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:958)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287)
> ... 7 more
> Caused by: java.io.IOException: Multiple family directories found in
> hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary
> at
> org.apache.hadoop.hive.hbase.HiveHFileOutputFormat$1.close(HiveHFileOutputFormat.java:158)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.closeWriters(FileSinkOperator.java:185)
> ... 11 more
> {code}
> The issue is that is looks for the HFiles in
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary}}
> when I believe it should be looking in the task attempt subfolder, such as
> {{hdfs://c1n1.gbif.org:8020/user/hive/warehouse/tim.db/coords_hbase/_temporary/2/_temporary/attempt_1461004169450_0002_r_00_1000}}.
> This can be reproduced in any HFile creation such as:
> {code:sql}
> CREATE TABLE coords_hbase(id INT, x DOUBLE, y DOUBLE)
> STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
> WITH SERDEPROPERTIES (
> 'hbase.columns.mapping' = ':key,o:x,o:y',
> 'hbase.table.default.storage.type' = 'binary');
> SET hfile.family.path=/tmp/coords_hfiles/o;
> SET hive.hbase.generatehfiles=true;
> INSERT OVERWRITE TABLE coords_hbase
> SELECT id, decimalLongitude, decimalLatitude
> FROM source
> CLUSTER BY id;
> {code}
> Any advice greatly appreciated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~ctang.ma] for the review. > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Fix For: 2.2.0 > > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675139#comment-15675139 ] Mohit Sabharwal edited comment on HIVE-15232 at 11/18/16 12:02 AM: --- Failures are unrelated to patch. Re-attaching to confirm. (Spark ones are failing with IllegalStateException: RPC channel is closed) was (Author: mohitsabharwal): Failures are unrelated to patch. (Spark ones are failing with IllegalStateException: RPC channel is closed) > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Attachment: HIVE-15232.2.patch > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675139#comment-15675139 ] Mohit Sabharwal commented on HIVE-15232: Failures are unrelated to patch. (Spark ones are failing with IllegalStateException: RPC channel is closed) > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Attachment: HIVE-15232.2.patch > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch, > HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674776#comment-15674776 ] Mohit Sabharwal commented on HIVE-15232: Addressed test failures. [~sushanth], [~ctang.ma], could you take a look ? Thanks. > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Attachment: HIVE-15232.2.patch > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch, HIVE-15232.2.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Status: Patch Available (was: Open) > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Attachment: HIVE-15232.1.patch > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > Attachments: HIVE-15232.1.patch > > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15232) Add notification events for functions and indexes
[ https://issues.apache.org/jira/browse/HIVE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-15232: --- Component/s: repl > Add notification events for functions and indexes > - > > Key: HIVE-15232 > URL: https://issues.apache.org/jira/browse/HIVE-15232 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Mohit Sabharwal >Assignee: Mohit Sabharwal > > Create/Drop Function and Create/Drop/Alter Index should also generate > metastore notification events. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15226) Add a different masking comment to qtests blobstore output
[ https://issues.apache.org/jira/browse/HIVE-15226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674344#comment-15674344 ] Mohit Sabharwal commented on HIVE-15226: LGTM +1 (I'm assuming addPatternWithMaskComment is called at most once for each unique pattern). > Add a different masking comment to qtests blobstore output > -- > > Key: HIVE-15226 > URL: https://issues.apache.org/jira/browse/HIVE-15226 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sergio Peña >Assignee: Sergio Peña >Priority: Minor > Attachments: HIVE-15226.1.patch, HIVE-15226.2.patch > > > The output of TestBlobstoreCliDriver is masking all s3a patch so that we can > use the tests with any other blobstore scheme. > It should be good to have a specific masking comment for those paths instead > of the generic " A masked pattern was here " so that we can verify > that certain tests are indeed using the blobstore path. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13966: --- Attachment: HIVE-13966.5.patch > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.4.patch, HIVE-13966.4.patch, > HIVE-13966.5.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13966: --- Attachment: HIVE-13966.4.patch > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.4.patch, HIVE-13966.4.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645839#comment-15645839 ] Mohit Sabharwal commented on HIVE-13966: [~alangates], [~sushanth], [~ctang.ma], could you please take a look at the latest patch ? I updated RB at https://reviews.apache.org/r/52800/ Filed HIVE-15145 for a related issue that I saw while I was looking at HiveAlterTable. > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.4.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13966: --- Attachment: HIVE-13966.4.patch > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.4.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal reassigned HIVE-13966: -- Assignee: Mohit Sabharwal (was: Rahul Sharma) > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-14911) Notification entry should not be written for failed events.
[ https://issues.apache.org/jira/browse/HIVE-14911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal resolved HIVE-14911. Resolution: Duplicate > Notification entry should not be written for failed events. > --- > > Key: HIVE-14911 > URL: https://issues.apache.org/jira/browse/HIVE-14911 > Project: Hive > Issue Type: Bug >Reporter: Sravya Tirukkovalur > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15061) Metastore types are sometimes case sensitive
[ https://issues.apache.org/jira/browse/HIVE-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622361#comment-15622361 ] Mohit Sabharwal commented on HIVE-15061: LGTM +1 (I double-checked other places where coltypes are set like StatObjectConverter - found that we are converting to lower case there already) > Metastore types are sometimes case sensitive > > > Key: HIVE-15061 > URL: https://issues.apache.org/jira/browse/HIVE-15061 > Project: Hive > Issue Type: Bug > Components: API >Affects Versions: 1.1.0 >Reporter: Thomas Tauber-Marshall >Assignee: Chaoyu Tang > Attachments: HIVE-15061.1.patch, HIVE-15061.1.patch, HIVE-15061.patch > > > Impala recently encountered an issue with the metastore > ([IMPALA-4260|https://issues.cloudera.org/browse/IMPALA-4260] ) where column > stats would get dropped when adding a column to a table. > The reason seems to be that Hive does a case sensitive check on the column > stats types during an "alter table" and expects the types to be all lower > case. This case sensitive check doesn't appear to happen when the stats are > set in the first place. > We're solving this on the Impala end by storing types in the metastore as all > lower case, but Hive's behavior here is very confusing. It should either > always be case sensitive, so that you can't create column stats with types > that Hive considers invalid, or it should never be case sensitive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14822: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, [~vihangk1]! > Add support for credential provider for jobs launched from Hiveserver2 > -- > > Key: HIVE-14822 > URL: https://issues.apache.org/jira/browse/HIVE-14822 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Fix For: 2.2.0 > > Attachments: HIVE-14822.01.patch, HIVE-14822.02.patch, > HIVE-14822.03.patch, HIVE-14822.05.patch, HIVE-14822.06.patch, > HIVE-14822.07.patch > > > When using encrypted passwords via the Hadoop Credential Provider, > HiveServer2 currently does not correctly forward enough information to the > job configuration for jobs to read those secrets. If your job needs to access > any secrets, like S3 credentials, then there's no convenient and secure way > to configure this today. > You could specify the decryption key in files like mapred-site.xml that > HiveServer2 uses, but this would place the encryption password on local disk > in plaintext, which can be a security concern. > To solve this problem, HiveServer2 should modify job configuration to include > the environment variable settings needed to decrypt the passwords. > Specifically, it will need to modify: > * For MR2 jobs: > ** yarn.app.mapreduce.am.admin.user.env > ** mapreduce.admin.user.env > * For Spark jobs: > ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD > ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD > HiveServer2 can get the decryption password from its own environment, the > same way it does for its own credential provider store today. > Additionally, it can be desirable for HiveServer2 to have a separate > encrypted password file than what is used by the job. HiveServer2 may have > secrets that the job should not have, such as the metastore database password > or the password to decrypt its private SSL certificate. It is also best > practices to have separate passwords on separate files. To facilitate this, > Hive will also accept: > * A configuration for a path to a credential store to use for jobs. This > should already be uploaded in HDFS. (hive.server2.job.keystore.location or a > better name) If this is not specified, then HS2 will simply use the value of > hadoop.security.credential.provider.path. > * An environment variable for the password to decrypt the credential store > (HIVE_JOB_KEYSTORE_PASSWORD or better). If this is not specified, then HS2 > will simply use the standard environment variable for decrypting the Hadoop > Credential Provider. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576579#comment-15576579 ] Mohit Sabharwal commented on HIVE-14822: LGTM, +1 > Add support for credential provider for jobs launched from Hiveserver2 > -- > > Key: HIVE-14822 > URL: https://issues.apache.org/jira/browse/HIVE-14822 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-14822.01.patch, HIVE-14822.02.patch, > HIVE-14822.03.patch, HIVE-14822.05.patch, HIVE-14822.06.patch, > HIVE-14822.07.patch > > > When using encrypted passwords via the Hadoop Credential Provider, > HiveServer2 currently does not correctly forward enough information to the > job configuration for jobs to read those secrets. If your job needs to access > any secrets, like S3 credentials, then there's no convenient and secure way > to configure this today. > You could specify the decryption key in files like mapred-site.xml that > HiveServer2 uses, but this would place the encryption password on local disk > in plaintext, which can be a security concern. > To solve this problem, HiveServer2 should modify job configuration to include > the environment variable settings needed to decrypt the passwords. > Specifically, it will need to modify: > * For MR2 jobs: > ** yarn.app.mapreduce.am.admin.user.env > ** mapreduce.admin.user.env > * For Spark jobs: > ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD > ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD > HiveServer2 can get the decryption password from its own environment, the > same way it does for its own credential provider store today. > Additionally, it can be desirable for HiveServer2 to have a separate > encrypted password file than what is used by the job. HiveServer2 may have > secrets that the job should not have, such as the metastore database password > or the password to decrypt its private SSL certificate. It is also best > practices to have separate passwords on separate files. To facilitate this, > Hive will also accept: > * A configuration for a path to a credential store to use for jobs. This > should already be uploaded in HDFS. (hive.server2.job.keystore.location or a > better name) If this is not specified, then HS2 will simply use the value of > hadoop.security.credential.provider.path. > * An environment variable for the password to decrypt the credential store > (HIVE_JOB_KEYSTORE_PASSWORD or better). If this is not specified, then HS2 > will simply use the standard environment variable for decrypting the Hadoop > Credential Provider. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14942: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~taoli-hwx]! > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0 > > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14942) HS2 UI: Canceled queries show up in "Open Queries"
[ https://issues.apache.org/jira/browse/HIVE-14942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573060#comment-15573060 ] Mohit Sabharwal commented on HIVE-14942: LGTM, +1. Looks like I missed this case in HIVE-13099 > HS2 UI: Canceled queries show up in "Open Queries" > -- > > Key: HIVE-14942 > URL: https://issues.apache.org/jira/browse/HIVE-14942 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14942.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15569884#comment-15569884 ] Mohit Sabharwal commented on HIVE-13966: Thanks [~alangates]! [~rahul9269] is not available to work on this patch, so one of us can take it over. Happy to take it over if you'd like. Couple quick comments: 1) Looks like changes to AlterHandler (and HiveAlterHandler) are not really needed ? The listener(s) are anyways getting invoked in HMSHandler.alter_table_core (after the alterHandler.alterTable call). So invocations in HiveAlterHandler seem to be duplicates. 2) Some cleanup items like lots of extra imports in HiveMetaStore.java and location of apache license in DummyTransactionalListener > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Rahul Sharma >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14822) Add support for credential provider for jobs launched from Hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-14822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549625#comment-15549625 ] Mohit Sabharwal commented on HIVE-14822: Left some questions and comments on RB. > Add support for credential provider for jobs launched from Hiveserver2 > -- > > Key: HIVE-14822 > URL: https://issues.apache.org/jira/browse/HIVE-14822 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-14822.01.patch, HIVE-14822.02.patch, > HIVE-14822.03.patch > > > When using encrypted passwords via the Hadoop Credential Provider, > HiveServer2 currently does not correctly forward enough information to the > job configuration for jobs to read those secrets. If your job needs to access > any secrets, like S3 credentials, then there's no convenient and secure way > to configure this today. > You could specify the decryption key in files like mapred-site.xml that > HiveServer2 uses, but this would place the encryption password on local disk > in plaintext, which can be a security concern. > To solve this problem, HiveServer2 should modify job configuration to include > the environment variable settings needed to decrypt the passwords. > Specifically, it will need to modify: > * For MR2 jobs: > ** yarn.app.mapreduce.am.admin.user.env > ** mapreduce.admin.user.env > * For Spark jobs: > ** spark.yarn.appMasterEnv.HADOOP_CREDSTORE_PASSWORD > ** spark.executorEnv.HADOOP_CREDSTORE_PASSWORD > HiveServer2 can get the decryption password from its own environment, the > same way it does for its own credential provider store today. > Additionally, it can be desirable for HiveServer2 to have a separate > encrypted password file than what is used by the job. HiveServer2 may have > secrets that the job should not have, such as the metastore database password > or the password to decrypt its private SSL certificate. It is also best > practices to have separate passwords on separate files. To facilitate this, > Hive will also accept: > * A configuration for a path to a credential store to use for jobs. This > should already be uploaded in HDFS. (hive.server2.job.keystore.location or a > better name) If this is not specified, then HS2 will simply use the value of > hadoop.security.credential.provider.path. > * An environment variable for the password to decrypt the credential store > (HIVE_JOB_KEYSTORE_PASSWORD or better). If this is not specified, then HS2 > will simply use the standard environment variable for decrypting the Hadoop > Credential Provider. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14864) Distcp is not called from MoveTask when src is a directory
[
https://issues.apache.org/jira/browse/HIVE-14864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15536842#comment-15536842
]
Mohit Sabharwal commented on HIVE-14864:
Does srcFS.getContentSummary(src).getLength() return total number of files in
the directory ?
I think this whole condition needs to be re-thought. Because AFAIK, DistCp
speeds up copies when multiple
files are involved. There is no advantage in DistCp'ing a single file, no
matter how big that file. Which means
HIVE_EXEC_COPYFILE_MAXSIZE does not make sense even for a file.
IOW, we need to look into DistCp'ing directories only, and possibly ones which
contain more than a threshold
number of files.
> Distcp is not called from MoveTask when src is a directory
> --
>
> Key: HIVE-14864
> URL: https://issues.apache.org/jira/browse/HIVE-14864
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> In FileUtils.java the following code does not get executed even when src
> directory size is greater than HIVE_EXEC_COPYFILE_MAXSIZE because
> srcFS.getFileStatus(src).getLen() returns 0 when src is a directory. We
> should use srcFS.getContentSummary(src).getLength() instead.
> {noformat}
> /* Run distcp if source file/dir is too big */
> if (srcFS.getUri().getScheme().equals("hdfs") &&
> srcFS.getFileStatus(src).getLen() >
> conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE)) {
> LOG.info("Source is " + srcFS.getFileStatus(src).getLen() + " bytes.
> (MAX: " + conf.getLongVar(HiveConf.ConfVars.HIVE_EXEC_COPYFILE_MAXSIZE) +
> ")");
> LOG.info("Launch distributed copy (distcp) job.");
> HiveConfUtil.updateJobCredentialProviders(conf);
> copied = shims.runDistCp(src, dst, conf);
> if (copied && deleteSource) {
> srcFS.delete(src, true);
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14775) Cleanup IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14775: --- Resolution: Fixed Status: Resolved (was: Patch Available) Test failures are unrelated to patch. Pushed to master. Thanks [~zsombor.klara] for the contribution! > Cleanup IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-14775.1.patch, HIVE-14775.patch > > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14775) Cleanup IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-14775: --- Summary: Cleanup IOException usage in Metrics APIs (was: Investigate IOException usage in Metrics APIs) > Cleanup IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-14775.1.patch, HIVE-14775.patch > > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14100) Adding a new logged_in_user() UDF which returns the user provided when connecting
[
https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-14100:
---
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks [~peter vary] for the contribution!
> Adding a new logged_in_user() UDF which returns the user provided when
> connecting
> -
>
> Key: HIVE-14100
> URL: https://issues.apache.org/jira/browse/HIVE-14100
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Beeline
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Fix For: 2.2.0
>
> Attachments: HIVE-14100.2.patch, HIVE-14100.2.patch,
> HIVE-14100.2.patch, HIVE-14100.patch
>
>
> There is an existing current_user() UDF which returns the user provided by
> the configured {{hive.security.authenticator.manager}}. This is often the
> same as the user provided on connection, but some cases, like
> HadoopDefaultAuthenticator this could be different.
> Some cases we need the logged in user independently of the configured
> authenticator, so a new UDF is created which provides this - returns the
> {{SessionState.get().getUserName()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[ https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533457#comment-15533457 ] Mohit Sabharwal commented on HIVE-14100: Thanks, [~pvary], LGTM, +1. Could you fix the jira title to say you're adding a new UDF called logged_in_user() ? Currently, it appears you are fixing current_user() > current_user() returns invalid information > -- > > Key: HIVE-14100 > URL: https://issues.apache.org/jira/browse/HIVE-14100 > Project: Hive > Issue Type: Bug > Components: Authentication, Beeline >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Attachments: HIVE-14100.2.patch, HIVE-14100.2.patch, > HIVE-14100.2.patch, HIVE-14100.patch > > > Using HadoopDeaultAuthenticator the current_user() returns the username of > the unix user running hiveservice2. > Using SessionStateUserAuthenticator the current_user returns the username > which is provided when the connection started. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[
https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530478#comment-15530478
]
Mohit Sabharwal commented on HIVE-14100:
Thanks [~pvary]], sounds good!
Your patch is getting user name from {{SessionState.get().getUserName()}}
HIVE-9143 is getting user name from {{SessionState.getUserFromAuthenticator()}}
which calls {{SessionStateUserAuthenticator.getUserName()}} which calls the
{{sessionState.getUserName()}} (same as your patch).
Just to confirm, you're saying that *if* SessionStateUserAuthenticator is the
not the default (which it is, in {{hiveServer2.cmd}} file ), then
other HiveAuthenticationProviders (like say HadoopDefaultAuthenticator), may
return user that runs the hiveserver2 process ?
> current_user() returns invalid information
> --
>
> Key: HIVE-14100
> URL: https://issues.apache.org/jira/browse/HIVE-14100
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Beeline
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14100.2.patch, HIVE-14100.2.patch,
> HIVE-14100.2.patch, HIVE-14100.patch
>
>
> Using HadoopDeaultAuthenticator the current_user() returns the username of
> the unix user running hiveservice2.
> Using SessionStateUserAuthenticator the current_user returns the username
> which is provided when the connection started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14775) Investigate IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15530411#comment-15530411 ] Mohit Sabharwal commented on HIVE-14775: [~zsombor.klara] please attach & submit your patch to trigger tests. Thanks! > Investigate IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14775) Investigate IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527063#comment-15527063 ] Mohit Sabharwal commented on HIVE-14775: A bunch of JMXExceptions can get thrown by MetricsMBeanImpl#getAttribute here: https://github.com/apache/hive/blob/master/common/src/java/org/apache/hadoop/hive/common/metrics/MetricsMBeanImpl.java#L50-L51 AttributeNotFoundException, MBeanException, ReflectionException are all types of JMXException > Investigate IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14100) current_user() returns invalid information
[
https://issues.apache.org/jira/browse/HIVE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526902#comment-15526902
]
Mohit Sabharwal commented on HIVE-14100:
Thanks, [~pvary]!
Could you fix the description to say SessionStateUserAuthenticator instead of
SessionStateAuthenticator ?
Also, I'm confused here, isn't the SessionStateUserAuthenticator authenticator
passed as the {{hive.security.authenticator.manager}} config when HS2 is
started in hiveServer2.cmd ? HIVE-9143 is getting the user from
SessionState.getUserFromAuthenticator, so isn't that udf already using
SessionStateUserAuthenticator ?
> current_user() returns invalid information
> --
>
> Key: HIVE-14100
> URL: https://issues.apache.org/jira/browse/HIVE-14100
> Project: Hive
> Issue Type: Bug
> Components: Authentication, Beeline
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Minor
> Attachments: HIVE-14100.2.patch, HIVE-14100.patch
>
>
> Using HadoopDeaultAuthenticator the current_user() returns the username of
> the unix user running hiveservice2.
> Using SessionStateAuthenticator the current_user returns the username which
> is provided when the connection started.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14775) Investigate IOException usage in Metrics APIs
[ https://issues.apache.org/jira/browse/HIVE-14775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526800#comment-15526800 ] Mohit Sabharwal commented on HIVE-14775: Thanks, [~zsombor.klara], this looks like a useful cleanup! Couple minor comments on RB. Looks ok to trickle JMException all the way to the top instead of wrapping it inside IOException since MetricsMBean extends a JMX interface. cc [~szehon] in case he has any comments > Investigate IOException usage in Metrics APIs > - > > Key: HIVE-14775 > URL: https://issues.apache.org/jira/browse/HIVE-14775 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, Metastore >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara > > A large number of metrics APIs seem to declare to throw IOExceptions > needlessly. (incrementCounter, decrementCounter etc.) > This is not only misleading but it fills up the code with unnecessary catch > blocks never to be reached. > We should investigate if these exceptions are thrown at all, and remove them > if it is truly unused. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513999#comment-15513999 ] Mohit Sabharwal commented on HIVE-14774: LGTM as well +1 > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14774) Canceling query using Ctrl-C in beeline might lead to stale locks
[ https://issues.apache.org/jira/browse/HIVE-14774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511956#comment-15511956 ] Mohit Sabharwal commented on HIVE-14774: [~ctang.ma], since locks are cleared in driver.destroy(), I was wondering how killing the query process is related to clearing locks. > Canceling query using Ctrl-C in beeline might lead to stale locks > - > > Key: HIVE-14774 > URL: https://issues.apache.org/jira/browse/HIVE-14774 > Project: Hive > Issue Type: Bug > Components: Locking >Reporter: Chaoyu Tang >Assignee: Chaoyu Tang > Attachments: HIVE-14774.patch > > > Terminating a running query using Ctrl-C in Beeline might lead to stale locks > since the process running the query might still be able to acquire the locks > but fail to release them after the query terminate abnormally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14011) MessageFactory is not pluggable
[ https://issues.apache.org/jira/browse/HIVE-14011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474501#comment-15474501 ] Mohit Sabharwal commented on HIVE-14011: instance is private and only accessible via getInstance, so this it should always default to org.apache.hive.hcatalog.messaging.json.JSONMessageFactory which is the current behavior LGTM, +1 pending tests cc [~alangates] in case he has comments. > MessageFactory is not pluggable > --- > > Key: HIVE-14011 > URL: https://issues.apache.org/jira/browse/HIVE-14011 > Project: Hive > Issue Type: Bug >Reporter: Sravya Tirukkovalur > Attachments: HIVE-14011.patch > > > Property "hcatalog.message.factory.impl.json" is available to use a custom > message factory implementation. Although it is not pluggable as > MessageFatcory is hardcoded to use JSONMessageFactory. > https://github.com/apache/hive/blob/26b5c7b56a4f28ce3eabc0207566cce46b29b558/hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java#L39 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14614) Insert overwrite local directory fails with IllegalStateException
[
https://issues.apache.org/jira/browse/HIVE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436106#comment-15436106
]
Mohit Sabharwal commented on HIVE-14614:
Nit: I'd just define a new HADOOP_LOCAL_FS_SCHEME and do
{code}
+isLocal = path.toUri().getScheme().equals(HADOOP_LOCAL_FS_SCHEME);
{code}
for readability.
Otherwise LGTM pending tests. [~spena] should take a look since he worked on
HIVE-14270.
> Insert overwrite local directory fails with IllegalStateException
> -
>
> Key: HIVE-14614
> URL: https://issues.apache.org/jira/browse/HIVE-14614
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-14614.2.patch
>
>
> insert overwrite local directory select * from table; fails with
> "java.lang.IllegalStateException: Cannot create staging directory" when the
> path sent to the getTempDirForPath(Path path) is a local fs path.
> This is a regression caused by the fix for HIVE-14270
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[
https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417553#comment-15417553
]
Mohit Sabharwal commented on HIVE-14342:
Thanks, [~ngangam], LGTM +1
> Beeline output is garbled when executed from a remote shell
> ---
>
> Key: HIVE-14342
> URL: https://issues.apache.org/jira/browse/HIVE-14342
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14342.2.patch, HIVE-14342.patch, HIVE-14342.patch
>
>
> {code}
> use default;
> create table clitest (key int, name String, value String);
> insert into table clitest values
> (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0");
> {code}
> then run a select query
> {code}
> # cat /tmp/select.sql
> set hive.execution.engine=mr;
> select key,name,value
> from clitest
> where value="1" limit 1;
> {code}
> Then run beeline via a remote shell, for example
> {code}
> $ ssh -l root "sudo -u hive beeline -u
> jdbc:hive2://localhost:1 -n hive -p hive --silent=true
> --outputformat=csv2 -f /tmp/select.sql"
> root@'s password:
> 16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
> module jar containing PrefixTreeCodec is not present. Continuing without it.
> nullkey,name,value
> 1,TRUE,1
> null
> $
> {code}
> In older releases that the output is as follows
> {code}
> $ ssh -l root "sudo -u hive beeline -u
> jdbc:hive2://localhost:1 -n hive -p hive --silent=true
> --outputformat=csv2 -f /tmp/run.sql"
> Are you sure you want to continue connecting (yes/no)? yes
> root@'s password:
> 16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
> module jar containing PrefixTreeCodec is not present. Continuing without it.
> key,name,value
> 1,TRUE,1
> $
> {code}
> The output contains nulls instead of blank lines. This is due to the use of
> -Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able
> to run beeline as a background process. But this is the unfortunate side
> effect of that fix.
> Running beeline in background also produces garbled output.
> {code}
> # beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true
> --outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 >
> /tmp/beeline.txt &
> # cat /tmp/beeline.txt
> null1,TRUE,1
> #
> {code}
> So I think the use of jline.UnsupportedTerminal should be documented but not
> used automatically by beeline under the covers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14227) Investigate invalid SessionHandle and invalid OperationHandle
[
https://issues.apache.org/jira/browse/HIVE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15394327#comment-15394327
]
Mohit Sabharwal commented on HIVE-14227:
SessionHandle is getting passed in most (all?) calls to HS2:
https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift
In TThreadPoolServer
(https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/server/TThreadPoolServer.java#L300)
the call order seems to be:
createContext -> processContext -> openSession (where we capture and set
sessionHandle in the context as context.setSessionHandle(sessionHandle) ->
deleteContext
Similarly, could we do for, say ExecuteStatement:
createContext -> processContext -> ExecuteStatement (where we capture and set
sessionHandle like above) -> deleteContext
IOW, you can attach the session to any connection for every request
individually. And keep a count of number of connections for every session which
you decrement in deleteContext. When it reaches 0, you can delete the session.
Am I missing something ?
> Investigate invalid SessionHandle and invalid OperationHandle
> -
>
> Key: HIVE-14227
> URL: https://issues.apache.org/jira/browse/HIVE-14227
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14227.1.patch
>
>
> There are the following warnings.
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-55]: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
> SessionHandle [1bc00251-64e9-4a95-acb7-a7f53f773528]
> at
> org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:318)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> {noformat}
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-1060]: Error closing operation:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=13d930dd-316c-4c09-9f44-fee5f483e73d]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:185)
> at
> org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15393132#comment-15393132
]
Mohit Sabharwal commented on HIVE-14296:
+1 as well
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,735 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
>
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386585#comment-15386585
]
Mohit Sabharwal commented on HIVE-14296:
Yeah, good to get rid of sessionCount
I think the current code is making no distinction between a connection & a
session.
MetricsConstant.OPEN_CONNECTIONS is getting increment/decremented
at connection level. But we are also closing the session when the connection
is detected to be closed/dropped (in deleteContext). Which implies that
connection and session
are the same thing from p.o.v. of the metrics, which seems fine.
Separately, looks like MetricsConstant.OPEN_CONNECTIONS is used in both
HS2 and HMS, which means this count includes both HS2 and HMS connections when
HMS is embedded in HS2. [~szehon] looks like we need to have a separate metric
for HMS connections ?
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386506#comment-15386506
]
Mohit Sabharwal commented on HIVE-14296:
I'm wondering if use of ThriftCLIService#sessionCount is redundant.
Shouldn't we be using SessionManager#getOpenSessionCount() count instead ?
ThriftBinaryCLIService#deleteContext is already closing the session which will
remove the value from SessionManager#handleToSession
So, it seems to me that ThriftCLIService#sessionCount is not telling us anything
that SessionManager#getOpenSessionCount() isn't already.
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
>
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15385164#comment-15385164
]
Mohit Sabharwal commented on HIVE-14251:
LGTM.
If i understand correctly, the only difference between isCommonTypeOf and
implicitConvertible is this line
{code}
// Allow implicit String to Double conversion
if (fromPg == PrimitiveGrouping.STRING_GROUP && to ==
PrimitiveCategory.DOUBLE) {
return true;
}
{code}
Wondering if it's easy to re-use implicitConvertible instead of duplicating the
code ? Maybe add a flag to the method
for String to Double conversion ?
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
