[
https://issues.apache.org/jira/browse/HDDS-1424?focusedWorklogId=236400&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-236400
]
ASF GitHub Bot logged work on HDDS-1424:
----------------------------------------
Author: ASF GitHub Bot
Created on: 02/May/19 16:30
Start Date: 02/May/19 16:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #726: HDDS-1424.
Support multi-container robot test execution
URL: https://github.com/apache/hadoop/pull/726#issuecomment-488741108
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|:----:|----------:|--------:|:--------|
| 0 | reexec | 80 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | dupname | 0 | No case conflicting files found. |
| 0 | shelldocs | 2 | Shelldocs was not available. |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| -1 | test4tests | 0 | The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 | mvninstall | 613 | trunk passed |
| +1 | compile | 263 | trunk passed |
| +1 | mvnsite | 0 | trunk passed |
| +1 | shadedclient | 977 | branch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 180 | trunk passed |
||| _ Patch Compile Tests _ |
| +1 | mvninstall | 565 | the patch passed |
| +1 | compile | 276 | the patch passed |
| +1 | javac | 276 | the patch passed |
| +1 | mvnsite | 0 | the patch passed |
| -1 | shellcheck | 4 | The patch generated 2 new + 0 unchanged - 1 fixed =
2 total (was 1) |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | xml | 2 | The patch has no ill-formed XML file. |
| +1 | shadedclient | 760 | patch has no errors when building and testing
our client artifacts. |
| +1 | javadoc | 174 | the patch passed |
||| _ Other Tests _ |
| -1 | unit | 238 | hadoop-hdds in the patch failed. |
| -1 | unit | 1206 | hadoop-ozone in the patch failed. |
| +1 | asflicense | 45 | The patch does not generate ASF License warnings. |
| | | 5635 | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests |
hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient |
| | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.web.client.TestOzoneClient |
| | hadoop.ozone.om.TestOmMetrics |
| | hadoop.ozone.container.ozoneimpl.TestOzoneContainer |
| | hadoop.hdds.scm.pipeline.TestPipelineClose |
| | hadoop.ozone.om.TestOzoneManagerConfiguration |
| | hadoop.ozone.om.TestOmBlockVersioning |
| | hadoop.hdds.scm.pipeline.TestNode2PipelineMap |
| |
hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline
|
| | hadoop.ozone.scm.TestXceiverClientManager |
| | hadoop.ozone.web.TestOzoneVolumes |
| | hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean |
| | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures |
| | hadoop.ozone.client.rpc.TestReadRetries |
| | hadoop.ozone.web.TestOzoneRestWithMiniCluster |
| | hadoop.ozone.TestMiniOzoneCluster |
| | hadoop.ozone.scm.TestSCMNodeManagerMXBean |
| | hadoop.ozone.client.rpc.TestCommitWatcher |
| | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException |
| | hadoop.ozone.om.TestOzoneManager |
| | hadoop.ozone.web.client.TestBuckets |
| | hadoop.ozone.om.TestOzoneManagerHA |
| | hadoop.ozone.TestMiniChaosOzoneCluster |
| | hadoop.ozone.scm.TestGetCommittedBlockLengthAndPutKey |
| | hadoop.ozone.scm.TestContainerSmallFile |
| | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.TestStorageContainerManager |
| | hadoop.ozone.scm.node.TestSCMNodeMetrics |
| | hadoop.ozone.scm.TestAllocateContainer |
| | hadoop.ozone.web.client.TestVolume |
| | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | Client=17.05.0-ce Server=17.05.0-ce base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/726 |
| Optional Tests | dupname asflicense shellcheck shelldocs compile javac
javadoc mvninstall mvnsite unit shadedclient xml |
| uname | Linux 6f224256abb3 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 6a42745 |
| Default Java | 1.8.0_191 |
| shellcheck |
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/artifact/out/diff-patch-shellcheck.txt
|
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/artifact/out/patch-unit-hadoop-hdds.txt
|
| unit |
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/testReport/ |
| Max. process+thread count | 3313 (vs. ulimit of 5500) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-726/6/console |
| versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 236400)
Time Spent: 2h (was: 1h 50m)
> Support multi-container robot test execution
> --------------------------------------------
>
> Key: HDDS-1424
> URL: https://issues.apache.org/jira/browse/HDDS-1424
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Reporter: Elek, Marton
> Assignee: Elek, Marton
> Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> The ./smoketest folder in the distribution package contains robotframework
> based test scripts to test the main behaviour of Ozone.
> The tests have two layers:
> 1. robot test definitions to execute commands and assert the results (on a
> given host machine)
> 2. ./smoketest/test.sh which starts/stops the docker-compose based
> environments AND execute the selected robot tests inside the right hosts
> The second one (test.sh) has some serious limitations:
> 1. all the tests are executed inside the same container (om):
> https://github.com/apache/hadoop/blob/5f951ea2e39ae4dfe554942baeec05849cd7d3c2/hadoop-ozone/dist/src/main/smoketest/test.sh#L89
> Some of the tests (ozonesecure-mr, ozonefs) may require the flexibility to
> execute different robot tests in different containers.
> 2. The definition of the global test set is complex and hard to understood.
> The current code is:
> {code}
> TESTS=("basic")
> execute_tests ozone "${TESTS[@]}"
> TESTS=("auditparser")
> execute_tests ozone "${TESTS[@]}"
> TESTS=("ozonefs")
> execute_tests ozonefs "${TESTS[@]}"
> TESTS=("basic")
> execute_tests ozone-hdfs "${TESTS[@]}"
> TESTS=("s3")
> execute_tests ozones3 "${TESTS[@]}"
> TESTS=("security")
> execute_tests ozonesecure .
> {code}
> For example for ozonesecure the TESTS is not used. And the usage of bash
> lists require additional complexity in the execute_tests function.
> I propose here a very lightweight refactor. Instead of including both the
> test definitions AND the helper methods in test.sh I would separate them.
> Let's put a test.sh to each of the compose directories. The separated test.sh
> can include common methods from a main shell script. For example:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm basic/basic.robot
> execute_robot_test scm s3
> stop_docker_env
> generate_report
> {code}
> This is a more clean and more flexible definition. It's easy to execute just
> this test (as it's saved to the compose/ozones3 directory. And it's more
> flexible.
> Other example, where multiple containers are used to execute tests:
> {code}
> source "$COMPOSE_DIR/../testlib.sh"
> start_docker_env
> execute_robot_test scm ozonefs/ozonefs.robot
> export OZONE_HOME=/opt/ozone
> execute_robot_test hadoop32 ozonefs/hadoopo3fs.robot
> execute_robot_test hadoop31 ozonefs/hadoopo3fs.robot
> stop_docker_env
> generate_report
> {code}
> With this separation the definition of the helper methods (eg.
> execute_robot_test or stop_docker_env) would also be simplified.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
