[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477447#comment-15477447
]
Andrew Purtell commented on HBASE-12721:
We are missing docs on how to build a topology image, or drop in topology from
this issue into a checkout of clusterdock. I don't want to use a canned
topology I can't change. Going back to the old version where things "just work"
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425317#comment-15425317
]
Dima Spivak commented on HBASE-12721:
-
Thanks, [~busbey]. I'll move those tasks to live under a new umbrella JIRA
having to do with integrating the topology into testing. Documentation is
blocked at the moment by the uncertainty of where we should be pushing Docker
images. Once that's resolved, I'll hammer out updates to the ref guide and the
RC scripts before taking a stab at Jenkins jobs to run {{hbase-it}} upstream
somewhere.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425307#comment-15425307
]
Sean Busbey commented on HBASE-12721:
-
Do the subtasks need to get promoted to full tasks, or closed out?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Fix For: 2.0.0
>
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425177#comment-15425177
]
Sean Busbey commented on HBASE-12721:
-
+1 from me.
we'll need to either fix or exclude the RAT pings, but I'll do that as I commit
it now.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425147#comment-15425147
]
Dima Spivak commented on HBASE-12721:
-
For anyone keeping score at home, the remaining Pylint warnings are caused by
YETUS-309 (which I just submitted a patch for over there). How's it look, guys?
Can we commit this?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423303#comment-15423303
]
Hadoop QA commented on HBASE-12721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 3s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 3s {color}
| {color:red} The patch generated 9 new + 0 unchanged - 0 fixed = 9 total (was
0) {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 13s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m 34s
{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-16 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823973/HBASE-12721_v7.patch |
| JIRA Issue | HBASE-12721 |
| Optional Tests | asflicense pylint shellcheck shelldocs |
| uname | Linux a6437a449803 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d5080e8 |
| shellcheck | v0.4.4 |
| pylint | v1.6.4 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/3111/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/3111/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-HBASE-Build/3111/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3111/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch,
> HBASE-12721_v7.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
>
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423263#comment-15423263
]
Hadoop QA commented on HBASE-12721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 3s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 3s {color}
| {color:red} The patch generated 10 new + 0 unchanged - 0 fixed = 10 total
(was 0) {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 10s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 13s
{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 11s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-16 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823969/HBASE-12721_v6.patch |
| JIRA Issue | HBASE-12721 |
| Optional Tests | asflicense pylint shellcheck shelldocs |
| uname | Linux 0fd99684a984 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d5080e8 |
| shellcheck | v0.4.4 |
| pylint | v1.6.4 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/3110/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/3110/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-HBASE-Build/3110/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3110/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-12721_v5.patch, HBASE-12721_v6.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15423184#comment-15423184
]
Hadoop QA commented on HBASE-12721:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 3s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} pylint {color} | {color:red} 0m 3s {color}
| {color:red} The patch generated 15 new + 0 unchanged - 0 fixed = 15 total
(was 0) {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 10s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 12s
{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-08-16 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12823959/HBASE-12721_v5.patch |
| JIRA Issue | HBASE-12721 |
| Optional Tests | asflicense pylint shellcheck shelldocs |
| uname | Linux a4e33b7794fd 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d5080e8 |
| shellcheck | v0.4.4 |
| pylint | v1.6.4 |
| pylint |
https://builds.apache.org/job/PreCommit-HBASE-Build/3108/artifact/patchprocess/diff-patch-pylint.txt
|
| hbaseprotoc |
https://builds.apache.org/job/PreCommit-HBASE-Build/3108/artifact/patchprocess/patch-hbaseprotoc-root.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-HBASE-Build/3108/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3108/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
> Attachments: HBASE-12721_v5.patch
>
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422083#comment-15422083
]
Sean Busbey commented on HBASE-12721:
-
I think it got missed because it isn't in "Patch Available" status.
I'm +1 and planning to push as soon as I have my local repo ready for patches.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15421896#comment-15421896
]
Dima Spivak commented on HBASE-12721:
-
Hey Docker-ers, any chance I can get one last review on this?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412788#comment-15412788
]
Dima Spivak commented on HBASE-12721:
-
Assuming you're running Docker 1.11+, to build HBase images from Git SHA
(defaults to using Hadoop 2.7.2):
{noformat}
source /dev/stdin <<< "$(curl -sL http://tiny.cloudera.com/clusterdock.sh)"
CLUSTERDOCK_TOPOLOGY_IMAGE=dimaspivak/clusterdock:apache_hbase_topology
clusterdock_run ./bin/build_cluster --namespace=dimaspivak apache_hbase
--hadoop-version=2.7.1
--hadoop-tarball=https://archive.apache.org/dist/hadoop/core/hadoop-2.7.1/hadoop-2.7.1.tar.gz
--hbase-version=andysCommit --hbase-git-commit
1ecb0fce342ee878cf96f7a3165007192bedb2ef
{noformat}
To start the cluster:
{noformat}
CLUSTERDOCK_TOPOLOGY_IMAGE=dimaspivak/clusterdock:apache_hbase_topology
clusterdock_run ./bin/start_cluster --namespace=dimaspivak apache_hbase
--hadoop-version=2.7.1 --hbase-version=andysCommit
--secondary-nodes='node-{2..5}'
{noformat}
To run ITBLL against the cluster:
{noformat}
clusterdock_ssh node-1.cluster
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList...
{noformat}
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412752#comment-15412752
]
Dima Spivak commented on HBASE-12721:
-
New review up on RB. Update removes the bulk of the framework from Apache HBase
but leaves behind what's needed to create an {{apache_hbase}} topology that can
be plugged into the framework. Since the original patches went up,
{{clusterdock.sh}} from the framework has also been updated to make it easier
to SSH into nodes. In short, now instead of needing to worry about private
keys, assuming you've started up a cluster with nodes
{{node-\{1..5\}.cluster}}, you can simply run
{noformat}
clusterdock_ssh node-1.cluster
{noformat}
to get an interactive terminal or
{noformat}
clusterdock_ssh node-1.cluster 'echo "list" | hbase shell -n'
{noformat}
to run a command against a running Docker-based cluster.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408105#comment-15408105
]
Andrew Purtell commented on HBASE-12721:
Makes sense.
bq. Would an example help anyone still following this?
Yes. An example of setting up with a compilation from git sha and running ITBLL
with this framework would be awesome :-)
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407318#comment-15407318
]
Sean Busbey commented on HBASE-12721:
-
makes sense. sorry to see the project not under asf governance, but I agree
that a central place would be better than our dev-support.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407250#comment-15407250
]
stack commented on HBASE-12721:
---
Sounds good to me as a stopgap to get this project moving along to the next
stage. +1.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407233#comment-15407233
]
Dima Spivak commented on HBASE-12721:
-
Since I last posted here, I've done a lot of thinking about the best way to
contribute this to HBase. I've always felt a bit funny about having the whole
{{clusterdock}} framework live in HBase proper, since only the {{apache_hbase}}
topology part has anything to do with HBase, so I think I've come up with a
better way to let us use this while not bloating {{dev-support}}. Long story
short, we've put up the {{clusterdock}} framework in [Cloudera's
GitHub|https://github.com/cloudera/clusterdock] and pushed images of the
framework into [Docker Hub|https://hub.docker.com/r/cloudera/clusterdock/]. The
framework now supports use of topologies that are not part of the actual
framework through Docker's data volume functionality. That is, instead of
needing to keep one copy of the framework for our Cloudera CDH stuff, and then
a duplicate copy in Apache HBase, I propose we let the framework live in our
Cloudera Docker registry, and then only commit the {{apache_hbase}} topology,
which can then be used with the framework.
Does that make sense? Would an example help anyone still following this?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15373157#comment-15373157
]
Sean Busbey commented on HBASE-12721:
-
(note to myself that I'm reviewing this on reviewboard)
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352173#comment-15352173
]
Dima Spivak commented on HBASE-12721:
-
Yeah, give it a shot. You can pass comma-separated list of directories into
{{\-\-data-directories}} and it'll set up YARN and HDFS properties accordingly.
As an example, my instances have SSDs mounted into {{/data}}, {{/data1}},
{{/data2}}, etc. My invocation to start up nodes which we use to run
{{hbase-it}} nightly is:
{noformat}
clusterdock_run ./bin/start_cluster --always-pull -n network${RANDOM}
apache_hbase \
--hbase-version=${HBASE_BRANCH} --secondary-nodes='node-{2..5}.internal' \
--data-directories='/data,/data1,/data2,/data3,/data4,/data5'
--start-services
{noformat}
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352160#comment-15352160
]
Andrew Purtell commented on HBASE-12721:
I'd be curious how to use --data-directories with 24 volumes on a d2.8xlarge.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352152#comment-15352152
]
Dima Spivak commented on HBASE-12721:
-
Woot. Alright, let me spend another day cleaning up the code based on what I
got from running it through Pylint and then I'll post instructions for how
you/[~busbey] can try pushing some images to the Apache HBase Bintray and put
it out for some wider testing. Thanks again for trying it out.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352150#comment-15352150
]
Andrew Purtell commented on HBASE-12721:
Yeah that works.
> I actually use this when running with storage-dense AWS instances (e.g.
> d2._xlarge) myself to let me do ITBLL runs with tons of data (say 10 TB+)
> without needing to deal with logical volumes and whatnot.
This is pretty much what I want to do as often as possible when making releases.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352135#comment-15352135
]
Dima Spivak commented on HBASE-12721:
-
So [~apurtell], I think this already works. There's an argument to the
{{apache_hbase}} topology called {{\-\-data-directories}} that lets you specify
locations on which to put YARN and HDFS stuff. I actually use this when running
with storage-dense AWS instances (e.g. {{d2._xlarge}}) myself to let me do
ITBLL runs with tons of data (say 10 TB+) without needing to deal with logical
volumes and whatnot. At the end, the utility functions that remove containers
(under {{./bin/housekeeping}}) also figure out which data directories are
mounted and clean them up, too. Would that work?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352127#comment-15352127
]
Andrew Purtell commented on HBASE-12721:
Never mind, docker doesn't like that very much
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352122#comment-15352122
]
Andrew Purtell commented on HBASE-12721:
[~dimaspivak], what are your thoughts on bundling local SSD storage aka
'ephemeral store' volumes into a single logical volume and mounting it on
/var/lib/docker/volumes ? This would mean that any docker volumes created while
launching clusters would not be persisted across instance stop and restart, but
on the other hand volumes can be used to back cluster data like namenode and
datanode storage. This is important for getting reasonable I/O latency and
variance.
I know Docker data volumes are meant to persist independent of the container’s
life cycle. We'd still have that as long as the EC2 instance is up, or simply
rebooted, but not if it is put into 'stopped' state and later started again.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15352119#comment-15352119
]
Andrew Purtell commented on HBASE-12721:
success!
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15346937#comment-15346937
]
Dima Spivak commented on HBASE-12721:
-
Just to provide an update for anyone interested, I'm putting the finishing
touches on a a new mega-patch version of this to go up on ReviewBoard that
would encompass the single commit that would go in (I originally split the RB
into 3 for easier digestion where I got some useful comments from [~asamir]
that I iterated on, thanks for that). Following [~ndimiduk]'s advice, I've run
the Python package through Pylint and so have a monster list of [mostly minor]
stuff to improve/knowingly ignore.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
> Components: build, community, documentation, test
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314637#comment-15314637
]
Dima Spivak commented on HBASE-12721:
-
Looks like the framework can handle what you want it to do, [~apurtell]. Here's
how:
# Log into your AWS host and create a file that looks like this:
{noformat}
root@trying-apache-docker:~# cat andy-setup.cfg
[hadoop/slaves]
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
[hadoop/core-site.xml]
fs.default.name = hdfs://{primary_node[0]}.{network}:8020
[hadoop/mapred-site.xml]
mapreduce.framework.name = yarn
[hadoop/yarn-site.xml]
yarn.resourcemanager.hostname = {primary_node[0]}.{network}
yarn.nodemanager.aux-services = mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce_shuffle.class =
org.apache.hadoop.mapred.ShuffleHandler
yarn.nodemanager.vmem-check-enabled = false
[hbase/regionservers]
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
[hbase/backup-masters]
{secondary_nodes[0]}.{network}
[hbase/hbase-site.xml]
hbase.cluster.distributed = true
hbase.rootdir = hdfs://{primary_node[0]}.{network}/hbase
hbase.zookeeper.quorum = {primary_node[0]}.{network}
hbase.zookeeper.property.dataDir = /usr/local/zookeeper
hbase.it.clustermanager.hadoop.hdfs.user = root
hbase.it.clustermanager.zookeeper.user = root
hbase.it.clustermanager.hbase.user = root
[hadoop/hadoop-env.sh]
body:
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+UseG1GC"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:MaxGCPauseMillis=100"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+PrintGCDetails"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+PrintGCDateStamps"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+PrintGCTimeStamps"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+PrintReferenceGC"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:+TieredCompilation"
COMMON_HDFS_OPTS="$COMMON_HDFS_OPTS -XX:-ResizePLAB"
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS $COMMON_HDFS_OPTS"
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_NAMENODE_OPTS="$HADOOP_NAMENODE_OPTS -verbose:gc
-Xloggc:/var/log/hadoop/hdfs-namenode-gc.log"
export HADOOP_SECONDARYNAMENODE_OPTS="$HADOOP_SECONDARYNAMENODE_OPTS -Xms1g
-Xmx1g"
export HADOOP_SECONDARYNAMENODE_OPTS="$HADOOP_SECONDARYNAMENODE_OPTS
$COMMON_HDFS_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="$HADOOP_SECONDARYNAMENODE_OPTS
-verbose:gc -Xloggc:/var/log/hadoop/hdfs-secondarynamenode-gc.log"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -Xms1g -Xmx1g"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS $COMMON_HDFS_OPTS"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_DATANODE_OPTS="$HADOOP_DATANODE_OPTS -verbose:gc
-Xloggc:/var/log/hadoop/hdfs-datanode-gc.log"
[hbase/hbase-env.sh]
body:
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+UseG1GC"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCDateStamps"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintGCTimeStamps"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+PrintReferenceGC"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:+TieredCompilation"
COMMON_HBASE_OPTS="$COMMON_HBASE_OPTS -XX:-ResizePLAB"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -Xms1g -Xmx1g"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS $COMMON_HBASE_OPTS"
export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-master-gc.log"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -Xms32g -Xmx32g"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS $COMMON_HBASE_OPTS"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS
-XX:MaxGCPauseMillis=50"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS
-XX:+UseCondCardMark"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS
-XX:+AlwaysPreTouch"
export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-regionserver-gc.log"
export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -Xms1g -Xmx1g"
export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS $COMMON_HBASE_OPTS"
export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -XX:+AlwaysPreTouch"
export HBASE_ZOOKEEPER_OPTS="$HBASE_ZOOKEEPER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-zookeeper-gc.log"
{noformat}
# Assuming the file is in your current working directory, run the following to
use it when starting up a
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313360#comment-15313360
]
Andrew Purtell commented on HBASE-12721:
bq. clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=0.98
--hbase-git-commit=0.98
Let me try this now. Really liking the ability to specify any git ref.
bq. Once those images exist, you can pass the location of an .ini-like file as
an argument to start_cluster and it will set configurations for you before
starting services
Ok, what I would like to do is control how hadoop-env.sh and hbase-env.sh are
generated. Consider something like:
*hadoop-env.sh*
{noformat}
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+UseG1GC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:MaxGCPauseMillis=100"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDetails"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCDateStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintGCTimeStamps"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+PrintReferenceGC"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:+TieredCompilation"
COMMON_HDFS_OPTS="\$COMMON_HDFS_OPTS -XX:-ResizePLAB"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -Xms1g -Xmx1g"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_NAMENODE_OPTS="\$HADOOP_NAMENODE_OPTS -verbose:gc
-Xloggc:/var/log/hadoop/hdfs-namenode-gc.log"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS -Xms1g
-Xmx1g"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS
\$COMMON_HDFS_OPTS"
export HADOOP_SECONDARYNAMENODE_OPTS="\$HADOOP_SECONDARYNAMENODE_OPTS
-verbose:gc -Xloggc:/var/log/hadoop/hdfs-secondarynamenode-gc.log"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -Xms1g -Xmx1g"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS \$COMMON_HDFS_OPTS"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -XX:+AlwaysPreTouch"
export HADOOP_DATANODE_OPTS="\$HADOOP_DATANODE_OPTS -verbose:gc
-Xloggc:/var/log/hadoop/hdfs-datanode-gc.log"
{noformat}
*hbase-env.sh*
{noformat}
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+UseG1GC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDetails"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCDateStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintGCTimeStamps"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintAdaptiveSizePolicy"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+PrintReferenceGC"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+ParallelRefProcEnabled"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:+TieredCompilation"
COMMON_HBASE_OPTS="\$COMMON_HBASE_OPTS -XX:-ResizePLAB"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -Xms1g -Xmx1g"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_MASTER_OPTS="\$HBASE_MASTER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-master-gc.log"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -Xms32g -Xmx32g"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS
-XX:MaxGCPauseMillis=50"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+UseCondCardMark"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -XX:+AlwaysPreTouch"
export HBASE_REGIONSERVER_OPTS="\$HBASE_REGIONSERVER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-regionserver-gc.log"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -Xms1g -Xmx1g"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS \$COMMON_HBASE_OPTS"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -XX:+AlwaysPreTouch"
export HBASE_ZOOKEEPER_OPTS="\$HBASE_ZOOKEEPER_OPTS -verbose:gc
-Xloggc:/var/log/hbase/hbase-zookeeper-gc.log"
{noformat}
That would handle all of the JVM flag wishlist I think.
bq. ./bin/build_cluster apache_hbase supports a flag that specifies a Java
tarball to use
Cool, I will try it.
bq. I imagine I'd need to modify the code a little bit to handle non-Oracle
releases, though.
Maybe not? The OpenJDK based JVM tarballs I build unpack like Oracle releases
and run fine on CentOS 6. They just lack JFR. I don't think you're using that.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313319#comment-15313319
]
Dima Spivak commented on HBASE-12721:
-
Yay! Glad to hear it works outside of my network :). As for the great points
raised:
bq. How does one build and register different versions of HBase for launching
by build_cluster? Possible to add to a local library?
You can do this for the 0.98 branch, for example, by running:
{noformat}
clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=0.98
--hbase-git-commit=0.98
{noformat}
In this case, the build process will check out source code, build a binary
tarball using Maven, and extract it into a proper Docker image that can then be
picked up by {{start_cluster}}. FYI, the {{hbase-version}} argument is just a
label for our internal use whereas {{hbase-git-commit}} is what's used when
checking out code; this lets you potentially do one-off builds of a particular
commit and name it whatever you want.
bq. Can we set up a library of HBase versions to test? 0.98, 1.1, 1.2, 1.3, and
master.
I hope so! We've had this running internally at Cloudera for a while where once
a night, we build these images, push them to our local repository, and then
assuming that that succeeds, trigger a Jenkins job that runs some tests from
{{hbase-it}}. Once we have the Docker registry part ironed out, I can
coordinate with a committer on setting up the necessary Jenkins infra needed to
do the same.
bq. Build_cluster should allow me to set Xms and Xmx, at least for
regionservers. If I start up a high memory instance I might want 8 GB heaps,
etc.
So the difference between {{build_cluster}} and {{start_cluster}} is that
{{build_cluster}} creates the necessary Docker image(s) needed for a
{{start_cluster}} to work properly. Once those images exist, you can pass the
location of an .ini-like file as an argument to {{start_cluster}} and it will
set configurations for you before starting services (and also keep them
synchronized among every node of your Docker-based cluster). Here's what the
default configuration looks like:
{noformat}
[hadoop/slaves]
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
[hadoop/core-site.xml]
fs.default.name = hdfs://{primary_node[0]}.{network}:8020
[hadoop/mapred-site.xml]
mapreduce.framework.name = yarn
[hadoop/yarn-site.xml]
yarn.resourcemanager.hostname = {primary_node[0]}.{network}
yarn.nodemanager.aux-services = mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce_shuffle.class =
org.apache.hadoop.mapred.ShuffleHandler
yarn.nodemanager.vmem-check-enabled = false
[hbase/regionservers]
+++ '\n'.join(["{{0}}.{network}".format(node) for node in {secondary_nodes}])
[hbase/backup-masters]
{secondary_nodes[0]}.{network}
[hbase/hbase-site.xml]
hbase.cluster.distributed = true
hbase.rootdir = hdfs://{primary_node[0]}.{network}/hbase
hbase.zookeeper.quorum = {primary_node[0]}.{network}
hbase.zookeeper.property.dataDir = /usr/local/zookeeper
hbase.it.clustermanager.hadoop.hdfs.user = root
hbase.it.clustermanager.zookeeper.user = root
hbase.it.clustermanager.hbase.user = root
{noformat}
Note that this file uses the group (thing inside [ ]) to decide which file to
modify, and it knows to format xml files as property files differently than
non-xml files. As such, we could just pass in JVM arguments in here.
bq. How would we use G1 instead of CMS? Bonus points for extra GC flag support
for Shenandoah.
Same as above, I think? If it can be controlled through an option in an
{{/hbase/conf}}-based file, this framework already supports it.
bq. How would we use a different version of the JVM? (including a custom
compiled version)
{{./bin/build_cluster apache_hbase}} supports a flag that specifies a Java
tarball to use. I imagine I'd need to modify the code a little bit to handle
non-Oracle releases, though.
bq. Let's add a script/wrapper that runs all of the IT tests. Extra credit if
one can optionally specify monkey type and policy on the command line.
+1!
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313292#comment-15313292
]
Andrew Purtell commented on HBASE-12721:
Awesome. I launched an Ubuntu 14.04 EC2 instance and followed your instructions
to get 1 primary + 5 secondary up and running in just a few minutes:
{noformat}
HBase Shell; enter 'help' for list of supported commands.
Type "exit" to leave the HBase Shell
Version 2.0.0-SNAPSHOT, rcbb95cd3a9bf9a9f8558560ae58f4061a73f15a8, Wed Jun 1
18:29:24 PDT 2016
hbase(main):001:0> status
1 active master, 1 backup masters, 5 servers, 0 dead, 0.4000 average load
{noformat}
Before I ran your setup script I did use LVM to make all locally attached SSD
storage into a single volume, formatted as XFS, and made /var/lib/docker a
symlink pointing to a directory on this device.
Questions and/or suggestions for enhancements follow. Perhaps subtasks? I'm
happy to help out as I'm able. The main point of interest for me is how to
build 0.98 SNAPSHOT images and run the IT test suite as part of release manager
duties.
- How does one build and register different versions of HBase for launching by
build_cluster? Possible to add to a local library?
- Can we set up a library of HBase versions to test? 0.98, 1.1, 1.2, 1.3, and
master.
- Build_cluster should allow me to set Xms and Xmx, at least for regionservers.
If I start up a high memory instance I might want 8 GB heaps, etc.
- How would we use G1 instead of CMS? Bonus points for extra GC flag support
for Shenandoah.
- How would we use a different version of the JVM? (including a custom compiled
version)
- Let's add a script/wrapper that runs all of the IT tests. Extra credit if one
can optionally specify monkey type and policy on the command line.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15313200#comment-15313200
]
Dima Spivak commented on HBASE-12721:
-
Hey [~vrodionov]. So what you're describing is definitely possible from a
technical standpoint (I actually got it working semi-decently in a recent
hackathon here at Cloudera), but I think for the time being, I'll focus on the
single-host use case to avoid some of the complications around keeping cluster
configuration files in sync.
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311632#comment-15311632
]
Vladimir Rodionov commented on HBASE-12721:
---
Great, definitely. One request: can you enhance your scripts and containerize
HBase in a cluster (not single), [~dimaspivak]? Run X HBase nodes per server in
M node cluster?
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12721) Create Docker container cluster infrastructure to enable better testing
[
https://issues.apache.org/jira/browse/HBASE-12721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311607#comment-15311607
]
Dima Spivak commented on HBASE-12721:
-
Please do! Here are some instructions to get started while I continue to hammer
out a proper patch:
# Install Docker 1.11 on a machine. I did the development and testing of this
under an Ubuntu 14.04 AWS instance with a 3.13.0-85 kernel (earlier Ubuntu
kernels might have problems), so something similar would be the surest way not
to run into issues. If you have a cloud instance at your disposal, you can
simplify setup by taking advantage of a framework script that I've temporarily
uploaded to Gist (I ran it through bit.ly because of the long raw URL, but I
won't be offended if you read through the script before running it). If using
the script, spin up your single cloud instance and then run {{curl -sL
http://bit.ly/20UntA8}}, which will install Docker, update the host's root
user's SSH keys to let you log into the containers we're going to start up, and
generally make your life easier. *Note that this script copies the SSH keys
from the Docker images we're creating into the {{~root/.ssh folder}} on your
host, so don't use this script on a machine with an existing SSH keypair in
{{~root/.ssh}}.*
# If you ran my setup script, run {{source /usr/local/bin/clusterdock.sh}} on
the host machine to enable use of {{clusterdock_run}}, a small helper function
that makes the framework super easy to use. _Instead of dealing with
virtualenvs for this Python project, we run it out of a Docker container_. If
you already had Docker installed and just want to source the script from Gist,
run
{noformat}
curl -sL http://bit.ly/1ROTmE3 | source /dev/stdin
{noformat}
# Stand up an HBase cluster. For the purpose of a quick demo, I uploaded images
I built with this framework to my personal Docker Hub acount. To get a running
4-node HBase cluster with Oracle Java 8u91, Hadoop 2.7.2, and the most recent
build of HBase {{master}} as of when I built it an hour ago, just type:
{noformat}
clusterdock_run ./bin/start_cluster apache_hbase --primary-node=node-1
--secondary-nodes='node-{2..4}' --hbase-version=master --start-services
{noformat}
That's it. Running that will pull down the necessary HBase image (also built
with this framework), setup some configs (i.e. you could have created a 5-node
cluster by changing the Bash-expandable argument to {{\-\-secondary-nodes}}),
and then start up services. Standard out will give information on what's
happening and how to access various web UIs, but you should also be able to SSH
to nodes by name (assuming you have the key-pairs setup) by using the hostnames
shown during startup. Other things to try:
- Stop and remove all running containers:
{noformat}
clusterdock_run ./bin/housekeeping nuke
{noformat}
- Build your own HBase cluster images:
{noformat}
clusterdock_run ./bin/build_cluster apache_hbase --hbase-version=master
--hbase-git-commit=master
{noformat}
- Learn more about options a particular script provides:
{noformat}
clusterdock_run ./bin/start_cluster -h
{noformat}
Let me know if you run into any issues!
> Create Docker container cluster infrastructure to enable better testing
> ---
>
> Key: HBASE-12721
> URL: https://issues.apache.org/jira/browse/HBASE-12721
> Project: HBase
> Issue Type: New Feature
>Reporter: Dima Spivak
>Assignee: Dima Spivak
>
> Some simple work on using HBase with Docker was committed into /dev-support
> as "hbase_docker;" all this did was stand up a standalone cluster from source
> and start a shell. Now seems like a good time to extend this to be useful for
> applications that could actual benefit the community, especially around
> testing. Some ideas:
> - Integration testing would be much more accessible if people could stand up
> distributed HBase clusters on a single host machine in a couple minutes and
> run our awesome hbase-it suite against it.
> - Binary compatibility testing of an HBase client is easiest when standing up
> an HBase cluster can be done once and then different client source/binary
> permutations run against it.
> - Upgrade testing, and especially rolling upgrade testing, doesn't have any
> upstream automation on build.apache.org, in part because it's a pain to set
> up x-node clusters on Apache infrastructure.
> This proposal, whether it stays under /dev-support or moves out into it's own
> top-level module ("hbase-docker" would conveniently fit the existing schema
> :-)), strives to create a simple framework for deploying "distributed,"
> multi-container Apache HBase clusters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
