[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[
https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456624#comment-15456624
]
Hive QA commented on HIVE-14589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12826677/HIVE-14589.04.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10442 tests
executed
*Failed tests:*
{noformat}
TestBeeLineWithArgs - did not produce a TEST-*.xml file
TestHiveCli - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1075/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1075/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1075/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12826677 - PreCommit-HIVE-MASTER-Build
> add consistent node replacement to LLAP for splits
> --
>
> Key: HIVE-14589
> URL: https://issues.apache.org/jira/browse/HIVE-14589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14589.01.patch, HIVE-14589.02.patch,
> HIVE-14589.03.patch, HIVE-14589.04.patch, HIVE-14589.patch
>
>
> See HIVE-14574. (copied from the comment below) This basically creates the
> nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest
> available slot, starting from 0. Unlike worker-... nodes, the slots are
> reused, which is the intent. The LLAPs are always sorted by the slot number
> for splits.
> The idea is that as long as LLAP is running, it will retain the same position
> in the ordering, regardless of other LLAPs restarting, without knowing about
> each other, the predecessors location (if restarted in a different place), or
> the total size of the cluster.
> The restarting LLAPs may not take the same positions as their predecessors
> (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter
> because they have lost their cache anyway.
> I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4,
> they will take whatever slots, but 3 will stay the 3rd and retain cache
> locality.
> This also handles size increase, as new LLAPs will always be added to the end
> of the sequence, which is what consistent hashing needs.
> One case it doesn't handle is permanent cluster size reduction. There will be
> a permanent gap if LLAPs are removed that have the slots in the middle; until
> some are restarted, it will result in misses
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[ https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456261#comment-15456261 ] Sergey Shelukhin commented on HIVE-14589: - Separating the create timeouts.. these can cause tests to fail to initialize. > add consistent node replacement to LLAP for splits > -- > > Key: HIVE-14589 > URL: https://issues.apache.org/jira/browse/HIVE-14589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14589.01.patch, HIVE-14589.02.patch, > HIVE-14589.03.patch, HIVE-14589.04.patch, HIVE-14589.patch > > > See HIVE-14574. (copied from the comment below) This basically creates the > nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest > available slot, starting from 0. Unlike worker-... nodes, the slots are > reused, which is the intent. The LLAPs are always sorted by the slot number > for splits. > The idea is that as long as LLAP is running, it will retain the same position > in the ordering, regardless of other LLAPs restarting, without knowing about > each other, the predecessors location (if restarted in a different place), or > the total size of the cluster. > The restarting LLAPs may not take the same positions as their predecessors > (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter > because they have lost their cache anyway. > I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4, > they will take whatever slots, but 3 will stay the 3rd and retain cache > locality. > This also handles size increase, as new LLAPs will always be added to the end > of the sequence, which is what consistent hashing needs. > One case it doesn't handle is permanent cluster size reduction. There will be > a permanent gap if LLAPs are removed that have the slots in the middle; until > some are restarted, it will result in misses -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[
https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15454212#comment-15454212
]
Hive QA commented on HIVE-14589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12826514/HIVE-14589.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10495 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1064/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1064/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1064/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12826514 - PreCommit-HIVE-MASTER-Build
> add consistent node replacement to LLAP for splits
> --
>
> Key: HIVE-14589
> URL: https://issues.apache.org/jira/browse/HIVE-14589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14589.01.patch, HIVE-14589.02.patch,
> HIVE-14589.03.patch, HIVE-14589.patch
>
>
> See HIVE-14574. (copied from the comment below) This basically creates the
> nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest
> available slot, starting from 0. Unlike worker-... nodes, the slots are
> reused, which is the intent. The LLAPs are always sorted by the slot number
> for splits.
> The idea is that as long as LLAP is running, it will retain the same position
> in the ordering, regardless of other LLAPs restarting, without knowing about
> each other, the predecessors location (if restarted in a different place), or
> the total size of the cluster.
> The restarting LLAPs may not take the same positions as their predecessors
> (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter
> because they have lost their cache anyway.
> I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4,
> they will take whatever slots, but 3 will stay the 3rd and retain cache
> locality.
> This also handles size increase, as new LLAPs will always be added to the end
> of the sequence, which is what consistent hashing needs.
> One case it doesn't handle is permanent cluster size reduction. There will be
> a permanent gap if LLAPs are removed that have the slots in the middle; until
> some are restarted, it will result in misses
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[ https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453588#comment-15453588 ] Sergey Shelukhin commented on HIVE-14589: - please ignore Host... stuff changes in RB. They are not in the patch, not sure where RB gets them > add consistent node replacement to LLAP for splits > -- > > Key: HIVE-14589 > URL: https://issues.apache.org/jira/browse/HIVE-14589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14589.01.patch, HIVE-14589.02.patch, > HIVE-14589.03.patch, HIVE-14589.patch > > > See HIVE-14574. (copied from the comment below) This basically creates the > nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest > available slot, starting from 0. Unlike worker-... nodes, the slots are > reused, which is the intent. The LLAPs are always sorted by the slot number > for splits. > The idea is that as long as LLAP is running, it will retain the same position > in the ordering, regardless of other LLAPs restarting, without knowing about > each other, the predecessors location (if restarted in a different place), or > the total size of the cluster. > The restarting LLAPs may not take the same positions as their predecessors > (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter > because they have lost their cache anyway. > I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4, > they will take whatever slots, but 3 will stay the 3rd and retain cache > locality. > This also handles size increase, as new LLAPs will always be added to the end > of the sequence, which is what consistent hashing needs. > One case it doesn't handle is permanent cluster size reduction. There will be > a permanent gap if LLAPs are removed that have the slots in the middle; until > some are restarted, it will result in misses -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[
https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450456#comment-15450456
]
Hive QA commented on HIVE-14589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12826233/HIVE-14589.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10472 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1046/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/1046/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-1046/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12826233 - PreCommit-HIVE-MASTER-Build
> add consistent node replacement to LLAP for splits
> --
>
> Key: HIVE-14589
> URL: https://issues.apache.org/jira/browse/HIVE-14589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14589.01.patch, HIVE-14589.02.patch,
> HIVE-14589.patch
>
>
> See HIVE-14574. (copied from the comment below) This basically creates the
> nodes in ZK for "slots" in the cluster. The LLAPs try to take the lowest
> available slot, starting from 0. Unlike worker-... nodes, the slots are
> reused, which is the intent. The LLAPs are always sorted by the slot number
> for splits.
> The idea is that as long as LLAP is running, it will retain the same position
> in the ordering, regardless of other LLAPs restarting, without knowing about
> each other, the predecessors location (if restarted in a different place), or
> the total size of the cluster.
> The restarting LLAPs may not take the same positions as their predecessors
> (i.e. if two LLAPs restart they can swap slots) but it shouldn't matter
> because they have lost their cache anyway.
> I.e. if you have LLAPs with slots 1-2-3-4 and I nuke and restart 1, 2, and 4,
> they will take whatever slots, but 3 will stay the 3rd and retain cache
> locality.
> This also handles size increase, as new LLAPs will always be added to the end
> of the sequence, which is what consistent hashing needs.
> One case it doesn't handle is permanent cluster size reduction. There will be
> a permanent gap if LLAPs are removed that have the slots in the middle; until
> some are restarted, it will result in misses
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[
https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436742#comment-15436742
]
Hive QA commented on HIVE-14589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12825379/HIVE-14589.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10447 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_join_part_col_char]
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3]
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/985/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/985/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-985/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12825379 - PreCommit-HIVE-MASTER-Build
> add consistent node replacement to LLAP for splits
> --
>
> Key: HIVE-14589
> URL: https://issues.apache.org/jira/browse/HIVE-14589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14589.01.patch, HIVE-14589.patch
>
>
> See HIVE-14574
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[ https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436060#comment-15436060 ] Sergey Shelukhin commented on HIVE-14589: - This basically creates the nodes in ZK for "slots" in the cluster. The nodes try to take the lowest available slot, starting from 0. Unlike worker-... nodes, the slots are reused, which is the intent. The nodes are always sort by the slot number for splits. The idea is that as long as the node is running, it will retain the same position in the ordering, regardless of other nodes restarting, without knowing about each other, their predecessors location, or the total count of nodes in the cluster. The restarting nodes may not take the same positions as their predecessors (i.e. if two nodes restart they can swap slots) but it doesn't matter as much because they have lost their cache anyway. I.e. if you have nodes 1-2-3-4 and I nuke and restart 1, 2, and 4, they will take whatever spots, but 3 will stay 3rd and retain cache locality. One case it doesn't handle is permanent cluster size reduction. There will be a permanent gap if nodes are removed that have the slots in the middle; until some nodes restart, it will result in misses. > add consistent node replacement to LLAP for splits > -- > > Key: HIVE-14589 > URL: https://issues.apache.org/jira/browse/HIVE-14589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14589.01.patch, HIVE-14589.patch > > > See HIVE-14574 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[ https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15436052#comment-15436052 ] Siddharth Seth commented on HIVE-14589: --- [~sershe] - could you provide a brief description of the change please. Makes the review a little easier. > add consistent node replacement to LLAP for splits > -- > > Key: HIVE-14589 > URL: https://issues.apache.org/jira/browse/HIVE-14589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-14589.01.patch, HIVE-14589.patch > > > See HIVE-14574 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14589) add consistent node replacement to LLAP for splits
[
https://issues.apache.org/jira/browse/HIVE-14589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432594#comment-15432594
]
Hive QA commented on HIVE-14589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12824966/HIVE-14589.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10479 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_mapjoin]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_2]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part1]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[load_dyn_part2]
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[transform_ppr1]
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching
org.apache.hive.service.cli.operation.TestOperationLoggingLayout.testSwitchLogLayout
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/962/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/962/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-962/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12824966 - PreCommit-HIVE-MASTER-Build
> add consistent node replacement to LLAP for splits
> --
>
> Key: HIVE-14589
> URL: https://issues.apache.org/jira/browse/HIVE-14589
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14589.patch
>
>
> See HIVE-14574
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
