[
https://issues.apache.org/jira/browse/LENS-893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203940#comment-15203940
]
Amareshwari Sriramadasu commented on LENS-893:
----------------------------------------------
We are seeing following exceptions in our production:
{noformat}
17 Mar 2016 08:58:58 [c6000f1d-f098-4b94-8298-36fcea70edf6] [StatusPoller]
ERROR org.apache.lens.server.query.QueryExecutionServiceImpl - Status update
failed for c6000f1d-f098-4b94-8298-36fcea70edf6
org.apache.lens.server.api.error.LensException: Internal Server Error.
at
org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:681)
~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
at
org.apache.lens.server.query.QueryExecutionServiceImpl.updateStatus(QueryExecutionServiceImpl.java:887)
[classes/:na]
at
org.apache.lens.server.query.QueryExecutionServiceImpl.access$1600(QueryExecutionServiceImpl.java:98)
[classes/:na]
at
org.apache.lens.server.query.QueryExecutionServiceImpl$StatusPoller.run(QueryExecutionServiceImpl.java:789)
[classes/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: org.apache.hive.service.cli.HiveSQLException: Unable to connect
after 3 retries
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.invoke(RetryingThriftCLIServiceClient.java:335)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at com.sun.proxy.$Proxy44.getOperationStatus(Unknown Source) ~[na:na]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient$CLIServiceClientWrapper.getOperationStatus(RetryingThriftCLIServiceClient.java:162)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:601)
~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
... 4 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Socket closed
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
... 8 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:203)
~[na:1.8.0_72]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_72]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
~[na:1.8.0_72]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
~[na:1.8.0_72]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
~[na:1.8.0_72]
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
... 14 common frames omitted
{noformat}
or
{noformat}
17 Mar 2016 08:49:55 [c6000f1d-f098-4b94-8298-36fcea70edf6] [StatusPoller]
ERROR org.apache.lens.driver.hive.HiveDriver - Error getting query status
org.apache.hive.service.cli.HiveSQLException: Unable to connect after 3 retries
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:241)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.invoke(RetryingThriftCLIServiceClient.java:335)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at com.sun.proxy.$Proxy44.getOperationStatus(Unknown Source) ~[na:na]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient$CLIServiceClientWrapper.getOperationStatus(RetryingThriftCLIServiceClient.java:162)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.lens.driver.hive.HiveDriver.updateStatus(HiveDriver.java:601)
~[lens-driver-hive-2.6.0-inm.jar:2.6.0-inm]
at
org.apache.lens.server.query.QueryExecutionServiceImpl.updateStatus(QueryExecutionServiceImpl.java:887)
[classes/:na]
at
org.apache.lens.server.query.QueryExecutionServiceImpl.access$1600(QueryExecutionServiceImpl.java:98)
[classes/:na]
at
org.apache.lens.server.query.QueryExecutionServiceImpl$StatusPoller.run(QueryExecutionServiceImpl.java:789)
[classes/:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: org.apache.thrift.transport.TTransportException:
java.net.SocketException: Socket closed
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:288)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connect(RetryingThriftCLIServiceClient.java:280)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
at
org.apache.hive.service.cli.thrift.RetryingThriftCLIServiceClient.connectWithRetry(RetryingThriftCLIServiceClient.java:237)
~[hive-service-0.13.4-inm.jar:0.13.4-inm]
... 8 common frames omitted
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.read(SocketInputStream.java:203)
~[na:1.8.0_72]
at java.net.SocketInputStream.read(SocketInputStream.java:141)
~[na:1.8.0_72]
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
~[na:1.8.0_72]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
~[na:1.8.0_72]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
~[na:1.8.0_72]
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127)
~[hive-exec-0.13.4-inm.jar:0.13.4-inm]
... 14 common frames omitted
{noformat}
Planning to add exponential back-off for next retry from lens status poller,
and getStatus API call would give last known status.
> Query Service should not mark queries failed in case driver is not responding
> in a timely manner
> ------------------------------------------------------------------------------------------------
>
> Key: LENS-893
> URL: https://issues.apache.org/jira/browse/LENS-893
> Project: Apache Lens
> Issue Type: Improvement
> Components: server
> Reporter: Rajat Khandelwal
> Assignee: Amareshwari Sriramadasu
> Fix For: 2.6
>
>
> Exponential backoff can be done in update status.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
