[
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437703#comment-15437703
]
Krystal commented on DRILL-4766:
--------------------------------
Used the following steps to re-produce problem without the fix:
1. Added 10000 schemas/tables to hive.
2. Lowered the rpc bit timeout (drillbit heart beat) to 10 seconds in
drill-override.conf (rpc.bit.timeout: 10).
3. Ran the following query:
select DISTINCT TABLE_TYPE from INFORMATION_SCHEMA.`TABLES` WHERE TABLE_TYPE
LIKE '%';
4. Cancelled the query.
5. Kicked off several queries from different clients.
The queries from Step 5 that connect to the same foreman as the cancelled query
consistently fail with error:
{code}
ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.
Connection: /10.10.100.123:60884 <--> /10.10.100.123:31011 (control client).
Closing connection.
java.io.IOException: syscall:read(...)() failed: Connection reset by peer
2016-08-09 14:50:10,285 [BitServer-3] ERROR
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
fragment query_id {
part1: 2906422873165293041
part2: 4085948525804402223
}
major_fragment_id: 2
minor_fragment_id: 1
on endpoint address: "qa-node111"
user_port: 31010
control_port: 31011
data_port: 31012
with org.apache.drill.exec.rpc.ChannelClosedException: Channel closed
/10.10.100.123:60884 <--> /10.10.100.123:31011..
2016-08-09 14:50:10,286 [BitServer-3] ERROR
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL
fragment query_id {
part1: 2906422873165293041
part2: 4085948525804402223
}
{code}
Executed the same steps against build with the fix and did not encounter any
failures.
> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -------------------------------------------------------------------------
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
> Issue Type: Improvement
> Components: Execution - Flow
> Affects Versions: 1.7.0
> Reporter: Deneche A. Hakim
> Assignee: Sudheesh Katkam
> Priority: Minor
> Fix For: 1.8.0
>
>
> Currently, rpc thread can block when trying to deliver a cancel or early
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios.
> FragmentExecutor could be improved to avoid blocking rpc threads as well
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
