[
https://issues.apache.org/jira/browse/LENS-348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336432#comment-14336432
]
Amareshwari Sriramadasu commented on LENS-348:
----------------------------------------------
Seems there is a deadlock between QuerySubmitter and Shutdown, both of them
trying to lock QueryExecutionServiceImpl object:
{noformat}
"QuerySubmitter" prio=10 tid=0x00007f2645db5800 nid=0x45da waiting for monitor
entry [0x00007f2655ec8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.lens.server.query.QueryExecutionServiceImpl.getMetrics(QueryExecutionServiceImpl.java:307)
- waiting to lock <0x00000000c09c5f58> (a
org.apache.lens.server.query.QueryExecutionServiceImpl)
at
org.apache.lens.server.query.QueryExecutionServiceImpl.incrCounter(QueryExecutionServiceImpl.java:332)
at
org.apache.lens.server.query.QueryExecutionServiceImpl.access$1000(QueryExecutionServiceImpl.java:71)
at
org.apache.lens.server.query.QueryExecutionServiceImpl$QuerySubmitter.run(QueryExecutionServiceImpl.java:481)
at java.lang.Thread.run(Thread.java:701)
"Shutdown" prio=10 tid=0x000000000249f800 nid=0x46ae in Object.wait()
[0x00007f263e6e5000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000c02649d0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1225)
- locked <0x00000000c02649d0> (a java.lang.Thread)
at java.lang.Thread.join(Thread.java:1278)
at
org.apache.lens.server.query.QueryExecutionServiceImpl.stop(QueryExecutionServiceImpl.java:902)
- locked <0x00000000c09c5f58> (a
org.apache.lens.server.query.QueryExecutionServiceImpl)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:102)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at
org.apache.hive.service.CompositeService.stop(CompositeService.java:92)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at org.apache.lens.server.LensServices.stop(LensServices.java:362)
- locked <0x00000000c0b6b858> (a org.apache.lens.server.LensServices)
at org.apache.lens.server.LensServer.stop(LensServer.java:159)
- locked <0x00000000c0b6b8b0> (a org.apache.lens.server.LensServer)
at org.apache.lens.server.LensServer$1.run(LensServer.java:259)
{noformat}
> Lens stop is in hung state when there is ReadTimeoutException with Hive
> Metastore
> ---------------------------------------------------------------------------------
>
> Key: LENS-348
> URL: https://issues.apache.org/jira/browse/LENS-348
> Project: Apache Lens
> Issue Type: Bug
> Reporter: Arshad Matin
> Attachments: stopIssue.txt
>
>
> LensServer.log
> {noformat}
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO org.apache.lens.server.LensServices
> - Persistence File system object close complete
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO
> org.apache.hive.service.AbstractService - Service:quota is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO
> org.apache.hive.service.AbstractService - Service:scheduler is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO
> org.apache.hive.service.AbstractService - Service:metastore is stopped.
> 25 Feb 2015 06:43:31,219 [Shutdown] INFO
> org.apache.hive.service.AbstractService - Service:query is stopped.
> 25 Feb 2015 06:43:37,609 [QuerySubmitter] ERROR
> org.apache.lens.server.query.QueryExecutionServiceImpl - Error launching
> query dacdd41b-9a39-4e3d-95fa-fe0fae93daa9
> org.apache.lens.api.LensException:
> org.apache.hive.service.cli.HiveSQLException: Unable to connect after 1
> retries
> at
> org.apache.lens.driver.hive.RemoteThriftConnection.getClient(RemoteThriftConnection.java:68)
> at
> org.apache.lens.driver.hive.HiveDriver.getClient(HiveDriver.java:746)
> at
> org.apache.lens.driver.hive.HiveDriver.executeAsync(HiveDriver.java:470)
> at
> org.apache.lens.server.query.QueryExecutionServiceImpl$QuerySubmitter.run(QueryExecutionServiceImpl.java:464)
> at java.lang.Thread.run(Thread.java:701)
> Caused by: org.apache.hive.service.cli.HiveSQLException: Unable to connect
> after 1 retries
> at
> org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241)
> at
> org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.newRetryingCLIServiceClient(RetryingThriftCLIServiceClient.java:227)
> at
> org.apache.lens.driver.hive.RemoteThriftConnection.getClient(RemoteThriftConnection.java:63)
> ... 4 more
> Caused by: org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
> at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> at
> org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280)
> at
> org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237)
> ... 6 more
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:146)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
> ... 12 more
> 25 Feb 2015 06:53:27,602 [HiveDriver-ConnectionExpiryThread] INFO
> org.apache.lens.driver.hive.HiveDriver - Closed connection:1
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
