[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

[Krystal (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437703#comment-15437703
 ] 

Krystal commented on DRILL-4766:


Used the following steps to re-produce problem without the fix:
1. Added 1 schemas/tables to hive.
2. Lowered the rpc bit timeout (drillbit heart beat) to 10 seconds in 
drill-override.conf (rpc.bit.timeout: 10).
3. Ran the following query:
select DISTINCT TABLE_TYPE from INFORMATION_SCHEMA.`TABLES` WHERE TABLE_TYPE 
LIKE '%';
4. Cancelled the query.
5. Kicked off several queries from different clients.

The queries from Step 5 that connect to the same foreman as the cancelled query 
consistently fail with error:
{code}
ERROR o.a.d.exec.rpc.RpcExceptionHandler - Exception in RPC communication.  
Connection: /10.10.100.123:60884 <--> /10.10.100.123:31011 (control client).  
Closing connection.
java.io.IOException: syscall:read(...)() failed: Connection reset by peer
2016-08-09 14:50:10,285 [BitServer-3] ERROR 
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL 
fragment query_id {
  part1: 2906422873165293041
  part2: 4085948525804402223
}
major_fragment_id: 2
minor_fragment_id: 1
 on endpoint address: "qa-node111"
user_port: 31010
control_port: 31011
data_port: 31012
 with org.apache.drill.exec.rpc.ChannelClosedException: Channel closed 
/10.10.100.123:60884 <--> /10.10.100.123:31011..
2016-08-09 14:50:10,286 [BitServer-3] ERROR 
o.a.d.exec.work.foreman.QueryManager - Failure while attempting to CANCEL 
fragment query_id {
  part1: 2906422873165293041
  part2: 4085948525804402223
}
{code}

Executed the same steps against build with the fix and did not encounter any 
failures.

> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.8.0
>
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

[Rahul Challapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15437420#comment-15437420
 ] 

Rahul Challapalli commented on DRILL-4766:
--

[~sudheeshkatkam] Can you explain how to reproduce this issue without the fix?

> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
> Fix For: 1.8.0
>
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15414052#comment-15414052
 ] 

ASF GitHub Bot commented on DRILL-4766:
---

Github user asfgit closed the pull request at:

https://github.com/apache/drill/pull/561


> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412372#comment-15412372
 ] 

ASF GitHub Bot commented on DRILL-4766:
---

Github user sudheeshkatkam commented on the issue:

https://github.com/apache/drill/pull/561
  
+1


> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Sudheesh Katkam
>Priority: Minor
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DRILL-4766) FragmentExecutor should use EventProcessor and avoid blocking rpc threads

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/DRILL-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15411212#comment-15411212
 ] 

ASF GitHub Bot commented on DRILL-4766:
---

GitHub user adeneche opened a pull request:

https://github.com/apache/drill/pull/561

DRILL-4766: FragmentExecutor should use EventProcessor and avoid bloc…

…king rpc threads

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/adeneche/incubator-drill DRILL-4766

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/drill/pull/561.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #561


commit 55b3a452bc0540ee9ca55f4f2a64ce795e334623
Author: adeneche 
Date:   2016-07-08T15:08:38Z

DRILL-4766: FragmentExecutor should use EventProcessor and avoid blocking 
rpc threads




> FragmentExecutor should use EventProcessor and avoid blocking rpc threads
> -
>
> Key: DRILL-4766
> URL: https://issues.apache.org/jira/browse/DRILL-4766
> Project: Apache Drill
>  Issue Type: Improvement
>  Components: Execution - Flow
>Affects Versions: 1.7.0
>Reporter: Deneche A. Hakim
>Assignee: Deneche A. Hakim
>Priority: Minor
>
> Currently, rpc thread can block when trying to deliver a cancel or early 
> termination message to a blocked fragment executor.
> Foreman already uses an EventProcessor to avoid such scenarios. 
> FragmentExecutor could be improved to avoid blocking rpc threads as well



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)