[jira] [Commented] (HIVE-16013) Fragments without locality can stack up on nodes
[
https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15884025#comment-15884025
]
Hive QA commented on HIVE-16013:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854554/HIVE-16013.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10261 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=140)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hive.service.server.TestHS2HttpServer.testContextRootUrlRewrite
(batchId=186)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3770/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3770/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3770/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854554 - PreCommit-HIVE-Build
> Fragments without locality can stack up on nodes
>
>
> Key: HIVE-16013
> URL: https://issues.apache.org/jira/browse/HIVE-16013
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Siddharth Seth
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16013.1.patch, HIVE-16013.2.patch
>
>
> When no locality information is provide, task requests can stack up on a node
> because of consistent no selection. When locality information is not provided
> we should fallback to random selection for better work distribution.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16013) Fragments without locality can stack up on nodes
[ https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15883621#comment-15883621 ] Siddharth Seth commented on HIVE-16013: --- +1 > Fragments without locality can stack up on nodes > > > Key: HIVE-16013 > URL: https://issues.apache.org/jira/browse/HIVE-16013 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-16013.1.patch, HIVE-16013.2.patch > > > When no locality information is provide, task requests can stack up on a node > because of consistent no selection. When locality information is not provided > we should fallback to random selection for better work distribution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16013) Fragments without locality can stack up on nodes
[
https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15880042#comment-15880042
]
Hive QA commented on HIVE-16013:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12854080/HIVE-16013.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10254 tests
executed
*Failed tests:*
{noformat}
TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out)
(batchId=235)
org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14]
(batchId=223)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=211)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3715/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3715/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3715/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12854080 - PreCommit-HIVE-Build
> Fragments without locality can stack up on nodes
>
>
> Key: HIVE-16013
> URL: https://issues.apache.org/jira/browse/HIVE-16013
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Siddharth Seth
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16013.1.patch
>
>
> When no locality information is provide, task requests can stack up on a node
> because of consistent no selection. When locality information is not provided
> we should fallback to random selection for better work distribution.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16013) Fragments without locality can stack up on nodes
[
https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879954#comment-15879954
]
Siddharth Seth commented on HIVE-16013:
---
{code}
+if (nodeInfo != null && nodeInfo.canAcceptTask()) {
{code}
This gets in the way of determining the next host to use. The requested host
will have canAcceptTask = false (since we already tried and failed a local
allocation). The requestedHostIdx will stay at -1, and we'll the first
available in the reduce allNodes assigned. (The loop to determine the next host
isn't serving any purpose after the patch - since the canAcceptTask check is
removed).
Should've been caught by a unit test :(
I'd say create a separate list for the randomAllocation vs the consistent
locality based allocation. random filtered by canAccept. Locality based would
be the complete list.
Unrelated to this specific jira: what happens in cases where a node is not
found in 'allNodes' during a consistent allocation (node went down between
split generation and actual execution of the query). requestedHostIdx will stay
at -1. Is this handled?
{code}
// no locality-requested, iterate the available hosts in consistent order from
the beginning
{code}
This comment needs to be fixed. We're not iterating the available hosts any
longer.
StackServlet.java - unrelated change?
Nit: Unused import in the Test
> Fragments without locality can stack up on nodes
>
>
> Key: HIVE-16013
> URL: https://issues.apache.org/jira/browse/HIVE-16013
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 2.2.0
>Reporter: Siddharth Seth
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16013.1.patch
>
>
> When no locality information is provide, task requests can stack up on a node
> because of consistent no selection. When locality information is not provided
> we should fallback to random selection for better work distribution.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16013) Fragments without locality can stack up on nodes
[ https://issues.apache.org/jira/browse/HIVE-16013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879389#comment-15879389 ] Prasanth Jayachandran commented on HIVE-16013: -- [~sseth] can you please take a look? > Fragments without locality can stack up on nodes > > > Key: HIVE-16013 > URL: https://issues.apache.org/jira/browse/HIVE-16013 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 2.2.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-16013.1.patch > > > When no locality information is provide, task requests can stack up on a node > because of consistent no selection. When locality information is not provided > we should fallback to random selection for better work distribution. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
