[
https://issues.apache.org/jira/browse/SOLR-18642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl updated SOLR-18642:
-------------------------------
Description:
Today we have BATS for testing {{bin/solr}} and realistic CLI type tests in
bash. We have nothing similar for Windows. SOLR-17508 explored Pester, but it
went nowhere.
This issue proposes *CATS* — a {*}C{*}ross-platform {*}A{*}utomated
{*}T{*}esting {*}S{*}ystem — a Java-native harness for shell testing, so such
tests run under JUnit like any other test, in parallel, on every platform. The
name follows BATS (Bash Automated Testing System) deliberately: same idea, one
letter and one platform restriction fewer. _CATS is a new invention of this
JIRA, not an external product._
h3. The DSL
Pure Java with convenience methods, so a ported BATS line reads almost
unchanged:
{code:java}
@Test
public void aNodeWeStartIsTheOnlyOneListed() {
env("SOLR_HEAP", "512m");
run("solr start");
run("solr create -c " + collName + " --shards 2")
.assertContains("2 shard(s)");
run("solr status --port ${SOLR_PORT}")
.assertContains("running on port");
run("solr stop --all");
run("solr status").assertContains("No Solr nodes are running");
}
{code}
Variable references in a command line are resolved by the harness, not by the
shell — which is what makes one line work under both, since bash expands
{{$NAME}} and cmd.exe does not, while cmd.exe wants {{%NAME%}} and bash does
not.
{{run()}} asserts that the command succeeded, so the happy path carries no
status boilerplate; a command expected to fail uses {{{}runFail(){}}}, which
asserts a non-zero status and hands back the result to assert on what it
printed.
h3. One source, both shells
Each test runs under bash on Linux and macOS, and under cmd.exe on Windows. A
platform-specific construct — a Unix-only tool, a pipe, command substitution —
is rejected at the call site with the fluent method that replaces it named in
the message, rather than silently passing on one platform. A test that
genuinely is one shell's carries {{{}@EnabledOnShell{}}}.
Suites run in parallel forks: each test class gets its own {{{}SOLR_HOME{}}},
its own {{SOLR_PID_DIR}} and its own block of ports.
A failure prints the command transcript — every command, its exit code and its
output — and one {{FAILED:}} line.
was:
Today we have BATS for testing {{bin/solr}} and realistic CLI type tests in
bash. We have nothing similar for Windows. SOLR-17508 explored Pester, but it
went nowhere.
This issue proposes *CATS* — a Cross-platform Automated Testing System — a
Java-native harness for shell testing, so such tests run under JUnit like any
other test, in parallel, on every platform. The name follows BATS (Bash
Automated Testing System) deliberately: same idea, one letter and one platform
restriction fewer.
h3. The DSL
Pure Java with convenience methods, so a ported BATS line reads almost
unchanged:
{code:java}
@Test
public void aNodeWeStartIsTheOnlyOneListed() {
env("SOLR_HEAP", "512m");
run("solr start");
run("solr create -c " + collName + " --shards 2")
.assertContains("2 shard(s)");
run("solr status --port ${SOLR_PORT}")
.assertContains("running on port");
run("solr stop --all");
run("solr status").assertContains("No Solr nodes are running");
}
{code}
Variable references in a command line are resolved by the harness, not by the
shell — which is what makes one line work under both, since bash expands
{{$NAME}} and cmd.exe does not, while cmd.exe wants {{%NAME%}} and bash does
not.
{{run()}} asserts that the command succeeded, so the happy path carries no
status boilerplate; a command expected to fail uses {{{}runFail(){}}}, which
asserts a non-zero status and hands back the result to assert on what it
printed.
h3. One source, both shells
Each test runs under bash on Linux and macOS, and under cmd.exe on Windows. A
platform-specific construct — a Unix-only tool, a pipe, command substitution —
is rejected at the call site with the fluent method that replaces it named in
the message, rather than silently passing on one platform. A test that
genuinely is one shell's carries {{{}@EnabledOnShell{}}}.
Suites run in parallel forks: each test class gets its own {{{}SOLR_HOME{}}},
its own {{SOLR_PID_DIR}} and its own block of ports.
A failure prints the command transcript — every command, its exit code and its
output — and one {{FAILED:}} line.
> Replace BATS with CATS
> ----------------------
>
> Key: SOLR-18642
> URL: https://issues.apache.org/jira/browse/SOLR-18642
> Project: Solr
> Issue Type: Improvement
> Components: cli, test-framework
> Reporter: Jan Høydahl
> Assignee: Jan Høydahl
> Priority: Major
> Labels: pull-request-available, test-framework, windows
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Today we have BATS for testing {{bin/solr}} and realistic CLI type tests in
> bash. We have nothing similar for Windows. SOLR-17508 explored Pester, but it
> went nowhere.
> This issue proposes *CATS* — a {*}C{*}ross-platform {*}A{*}utomated
> {*}T{*}esting {*}S{*}ystem — a Java-native harness for shell testing, so such
> tests run under JUnit like any other test, in parallel, on every platform.
> The name follows BATS (Bash Automated Testing System) deliberately: same
> idea, one letter and one platform restriction fewer. _CATS is a new invention
> of this JIRA, not an external product._
> h3. The DSL
> Pure Java with convenience methods, so a ported BATS line reads almost
> unchanged:
> {code:java}
> @Test
> public void aNodeWeStartIsTheOnlyOneListed() {
> env("SOLR_HEAP", "512m");
> run("solr start");
> run("solr create -c " + collName + " --shards 2")
> .assertContains("2 shard(s)");
> run("solr status --port ${SOLR_PORT}")
> .assertContains("running on port");
> run("solr stop --all");
> run("solr status").assertContains("No Solr nodes are running");
> }
> {code}
> Variable references in a command line are resolved by the harness, not by the
> shell — which is what makes one line work under both, since bash expands
> {{$NAME}} and cmd.exe does not, while cmd.exe wants {{%NAME%}} and bash does
> not.
> {{run()}} asserts that the command succeeded, so the happy path carries no
> status boilerplate; a command expected to fail uses {{{}runFail(){}}}, which
> asserts a non-zero status and hands back the result to assert on what it
> printed.
> h3. One source, both shells
> Each test runs under bash on Linux and macOS, and under cmd.exe on Windows. A
> platform-specific construct — a Unix-only tool, a pipe, command substitution
> — is rejected at the call site with the fluent method that replaces it named
> in the message, rather than silently passing on one platform. A test that
> genuinely is one shell's carries {{{}@EnabledOnShell{}}}.
> Suites run in parallel forks: each test class gets its own {{{}SOLR_HOME{}}},
> its own {{SOLR_PID_DIR}} and its own block of ports.
> A failure prints the command transcript — every command, its exit code and
> its output — and one {{FAILED:}} line.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
