[
https://issues.apache.org/jira/browse/HADOOP-17191?focusedWorklogId=506990&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-506990
]
ASF GitHub Bot logged work on HADOOP-17191:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 03/Nov/20 14:08
Start Date: 03/Nov/20 14:08
Worklog Time Spent: 10m
Work Description: snvijaya commented on a change in pull request #2417:
URL: https://github.com/apache/hadoop/pull/2417#discussion_r515741344
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -357,6 +357,34 @@ The Huge File tests validate Azure storages's ability to
handle large files —t
Tests at this scale are slow: they are best executed from hosts running in
the cloud infrastructure where the storage endpoint is based.
+##No test no review: Run different combinations of tests using the runtests.sh
script
+
+This is the expected way in which the tests have to be ran before raising a PR.
Review comment:
Change to:
To simplify the testing across various authentication and features
combinations that are mandatory for a PR, script
`dev-support/testrun-scripts/runTest.sh` should be used. Once the script is
updated with relevant config settings for various test combinations, it will:
1. Auto-generate configs specific to each test combinations
2. Run tests for all combinations
3. Summarize results across all the test combination runs.
As a pre-requiste step, fill config values for test accounts and credentials
needed for authentication in `src/test/resources/azure-auth-keys.xml.template`
and rename as `src/test/resources/azure-auth-keys.xml`.
**To add a new test combination**: Mandatory test combinations for PR
validation are already pre-filled in `dev-support/testrun-scripts/runTest.sh`.
If a new one needs to be added, add a combination set within
`dev-support/testrun-scripts/runTest.sh` similar to the ones already defined and
1. Provide a new combination name
2. Update properties and values array which need to be effective for the
test combination
3. Call generateConfigs
**To run PR validation**: Running command
- `runTest.sh -testCombination #combinationname#` or `runTest.sh -tc
#combinationname#` : will generate configurations effective for that test
combination and also run the test.
- `runTest.sh -testCombination all` or `runTest.sh -tc all` : will generate
configurations for each of the combinations defined and run tests.
**Test logs**: <Put in the para starting "Once the tests are completed",
with the last line modified as :
When run for PR validation, the consolidated test results needs to be pasted
into the PR comment section.>
**To generate config for use in IDE**: Running command
`runTest.sh -generateConfig #combinationname#` or `runTest.sh -gc
#combinationname#`
will update the effective config relevant for the specific test combination.
Hence the same config files used by the mvn test runs can be used for IDE
without any manual updates needed within config file.
**Other command line options**:
1. Thread count : ABFS mvn tests are run in parallel mode. Tests by default
are run with 8 thread count. It can be changed by providing -t #ThreadCount#
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -357,6 +357,34 @@ The Huge File tests validate Azure storages's ability to
handle large files —t
Tests at this scale are slow: they are best executed from hosts running in
the cloud infrastructure where the storage endpoint is based.
+##No test no review: Run different combinations of tests using the runtests.sh
script
Review comment:
Change to
#### Generating test run configurations and test triggers over various
config combinations
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -357,6 +357,34 @@ The Huge File tests validate Azure storages's ability to
handle large files —t
Tests at this scale are slow: they are best executed from hosts running in
the cloud infrastructure where the storage endpoint is based.
+##No test no review: Run different combinations of tests using the runtests.sh
script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script `runtests.sh` contain template for 3 combinations of tests. Ensure
+the auth configs for all the accounts used for testing are provided in
+azure-auth-keys.xml. In case any new flags or properties are introduced
+with the code change, add the combinations with the possible configurations
+into the `runtests.sh`. The thread count can be specified as the command line
+argument for the script. By default the same will be 8. -n option can be
+specified if build is not required prior to the tests.
+
+Adding a combination of tests involves setting the variable combination (ex:
HNS
+-OAuth) and specifying the specific configurations for the particular
+combination with 2 arrays namely properties and values. Specify the property
+names within the array properties and corresponding values in the values
+array. The property and value is determined by the array index. The value for
+the property mentioned at index 1 of array properties should be specified at
+index 1 of the array values. Call the function generateconfigs once the 3
+values mentioned are set. Now the script `runtests.sh` is ready to be ran.
+
+Once the tests are completed, logs will be present in the directory
+dev-support/testrun-scripts/testlogs/$startname, $startname will be the start
Review comment:
Better to have testLogs folder within dev-support than within script
folder.
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -357,6 +357,34 @@ The Huge File tests validate Azure storages's ability to
handle large files —t
Tests at this scale are slow: they are best executed from hosts running in
the cloud infrastructure where the storage endpoint is based.
+##No test no review: Run different combinations of tests using the runtests.sh
script
+
+This is the expected way in which the tests have to be ran before raising a PR.
Review comment:
Actually rewrite the Test logs section as below:
Test runs will create a folder within `dev-support/testlogs` to save the
test logs. Folder name will be the test start timestamp. The mvn test command
line logs for each combination will be saved into a file as `Test-Logs
-$combination` into this folder. Incase of any failures, this file will have
the failure exception stack.
At the end of the test run, the consolidated results of all the combination
runs will be saved into a file as `Test-Results.log` in the same folder. When
run for PR validation, the consolidated test results needs to be pasted into
the PR comment section.
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,224 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+testresourcesdir=src/test/resources
+combconfsdir=$testresourcesdir/combconfs
+combtestfile=$testresourcesdir/abfs-combination-test-configs.xml
+
+shouldbuild=1
+
+logdir=dev-support/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+properties=
+values=
+
+validate() {
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$combination" ]; then
+ echo "Exiting. combination cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for combination $combination"
+ exit -1
+ fi
+
+ for filename in "${combinations[@]}"; do
+ if [[ ! -f "$combconfsdir/$filename.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ done
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+cleancombinationconfigs() {
+ rm -rf $combconfsdir
+ mkdir -p $combconfsdir
+}
+
+generateconfigs() {
+ combconffile="$combconfsdir/$combination.xml"
+ rm -rf "$combconffile"
+ cat > "$combconffile" << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $combination"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ formatxml "$combconffile"
+}
+
+formatxml() {
+ xmlstarlet fo -s 2 "$1" > "$1.tmp"
+ mv "$1.tmp" "$1"
+}
+
+setactiveconf() {
+ if [[ ! -f "$combconfsdir/$combination.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ rm -rf $combtestfile
+ cat > $combtestfile << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+ xmlstarlet ed -P -L -s /configuration -t elem -n include -v "" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n href -v
"combconfs/$combination.xml" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n xmlns -v
"http://www.w3.org/2001/XInclude"; $combtestfile
+ formatxml $combtestfile
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" "$combconffile"
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property "$combconffile"
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n
value -v "$2" "$combconffile"
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+summary() {
+ {
+ echo ""
+ echo "$combination"
+ echo "========================"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+ } >> "$testresultsfilename"
+ printf "\n----- Test results -----\n"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+
+ secondstaken=$((ENDTIME - STARTTIME))
+ mins=$((secondstaken / 60))
+ secs=$((secondstaken % 60))
+ printf "\nTime taken: %s mins %s secs.\n" "$mins" "$secs"
+ echo "Find test logs for the combination ($combination) in: $testlogfilename"
+ echo "Find consolidated test results in: $testresultsfilename"
+ echo "----------"
+}
+
+init() {
+ checkdependencies
+ if [[ "$shouldbuild" -eq "1" ]]; then
+ if ! mvn clean install -DskipTests
+ then
+ echo ""
+ echo "Exiting. Build failed."
+ exit -1
+ fi
+ fi
+ starttime=$(date +"%Y-%m-%d_%H-%M-%S")
+ mkdir -p "$logdir"
+ testresultsfilename="$logdir/$starttime/Test-Results.txt"
+ if [[ -z "$combinations" ]]; then
+ combinations=( $( ls $combconfsdir/*.xml ))
+ fi
+}
+
+runtests() {
+ parseoptions "$@"
+ validate
+ if [ -z "$starttime" ]; then
+ init
+ fi
+ shopt -s nullglob
+ for combconffile in "${combinations[@]}"; do
+ STARTTIME=$(date +%s)
+ combination=$(basename "$combconffile" .xml)
+ mkdir -p "$logdir/$starttime"
+ testlogfilename="$logdir/$starttime/Test-Logs-$combination.txt"
+ printf "\nRunning the combination: %s..." "$combination"
+ setactiveconf
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount
verify >> "$testlogfilename" || true
+ ENDTIME=$(date +%s)
+ summary
+ done
+}
+
+begin() {
+ cleancombinationconfigs
+}
+
+parseoptions() {
+ while getopts ":c:a:n?t:" option; do
+ case "${option}" in
+ a)
+ combination=$(basename "$OPTARG" .xml)
+ setactiveconf
+ exit 0
+ ;;
+ c)
+ combination=$(basename "$OPTARG" .xml)
+ combinations+=("$combination")
+ ;;
+ n)
+ shouldbuild=0
+ ;;
+ t)
+ threadcount=$OPTARG
+ ;;
+ *|?|h)
+ echo "Usage: $0 [-n] [-a COMBINATION_NAME] [-c COMBINATION_NAME] [-t
THREAD_COUNT]"
+ echo ""
+ echo "Where:"
+ echo " -a COMBINATION_NAME Specify the combination name which needs
to be activated."
+ echo " -c COMBINATION_NAME Specify the combination name for test
runs"
+ echo " -n Specify this option if there is no need to build before
running the tests"
+ echo " -t THREAD_COUNT Specify the thread count"
+ exit 1
Review comment:
Had some suggestion on how the options to this script should be. Its
added in the documentation review comments. Have a look and we can discuss.
##########
File path: hadoop-tools/hadoop-azure/.gitignore
##########
@@ -1,2 +1,5 @@
.checkstyle
-bin/
\ No newline at end of file
+bin/
+src/test/resources/combconfs
Review comment:
Rename to combinationConfigFiles
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,224 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+testresourcesdir=src/test/resources
+combconfsdir=$testresourcesdir/combconfs
+combtestfile=$testresourcesdir/abfs-combination-test-configs.xml
+
+shouldbuild=1
+
+logdir=dev-support/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+properties=
+values=
+
+validate() {
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$combination" ]; then
+ echo "Exiting. combination cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for combination $combination"
+ exit -1
+ fi
+
+ for filename in "${combinations[@]}"; do
+ if [[ ! -f "$combconfsdir/$filename.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ done
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+cleancombinationconfigs() {
+ rm -rf $combconfsdir
+ mkdir -p $combconfsdir
+}
+
+generateconfigs() {
+ combconffile="$combconfsdir/$combination.xml"
+ rm -rf "$combconffile"
+ cat > "$combconffile" << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $combination"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ formatxml "$combconffile"
+}
+
+formatxml() {
+ xmlstarlet fo -s 2 "$1" > "$1.tmp"
+ mv "$1.tmp" "$1"
+}
+
+setactiveconf() {
+ if [[ ! -f "$combconfsdir/$combination.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ rm -rf $combtestfile
+ cat > $combtestfile << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+ xmlstarlet ed -P -L -s /configuration -t elem -n include -v "" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n href -v
"combconfs/$combination.xml" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n xmlns -v
"http://www.w3.org/2001/XInclude"; $combtestfile
+ formatxml $combtestfile
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" "$combconffile"
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property "$combconffile"
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n
value -v "$2" "$combconffile"
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+summary() {
+ {
+ echo ""
+ echo "$combination"
+ echo "========================"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+ } >> "$testresultsfilename"
+ printf "\n----- Test results -----\n"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+
+ secondstaken=$((ENDTIME - STARTTIME))
+ mins=$((secondstaken / 60))
+ secs=$((secondstaken % 60))
+ printf "\nTime taken: %s mins %s secs.\n" "$mins" "$secs"
+ echo "Find test logs for the combination ($combination) in: $testlogfilename"
+ echo "Find consolidated test results in: $testresultsfilename"
+ echo "----------"
+}
+
+init() {
+ checkdependencies
+ if [[ "$shouldbuild" -eq "1" ]]; then
+ if ! mvn clean install -DskipTests
+ then
+ echo ""
+ echo "Exiting. Build failed."
+ exit -1
+ fi
+ fi
+ starttime=$(date +"%Y-%m-%d_%H-%M-%S")
+ mkdir -p "$logdir"
+ testresultsfilename="$logdir/$starttime/Test-Results.txt"
+ if [[ -z "$combinations" ]]; then
+ combinations=( $( ls $combconfsdir/*.xml ))
+ fi
+}
+
+runtests() {
+ parseoptions "$@"
+ validate
+ if [ -z "$starttime" ]; then
+ init
+ fi
+ shopt -s nullglob
+ for combconffile in "${combinations[@]}"; do
+ STARTTIME=$(date +%s)
+ combination=$(basename "$combconffile" .xml)
+ mkdir -p "$logdir/$starttime"
+ testlogfilename="$logdir/$starttime/Test-Logs-$combination.txt"
+ printf "\nRunning the combination: %s..." "$combination"
+ setactiveconf
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount
verify >> "$testlogfilename" || true
+ ENDTIME=$(date +%s)
+ summary
+ done
+}
+
+begin() {
+ cleancombinationconfigs
+}
+
+parseoptions() {
+ while getopts ":c:a:n?t:" option; do
+ case "${option}" in
+ a)
+ combination=$(basename "$OPTARG" .xml)
+ setactiveconf
+ exit 0
+ ;;
+ c)
+ combination=$(basename "$OPTARG" .xml)
+ combinations+=("$combination")
+ ;;
+ n)
+ shouldbuild=0
+ ;;
+ t)
+ threadcount=$OPTARG
+ ;;
+ *|?|h)
+ echo "Usage: $0 [-n] [-a COMBINATION_NAME] [-c COMBINATION_NAME] [-t
THREAD_COUNT]"
+ echo ""
+ echo "Where:"
+ echo " -a COMBINATION_NAME Specify the combination name which needs
to be activated."
+ echo " -c COMBINATION_NAME Specify the combination name for test
runs"
Review comment:
What is the difference between -a and -c ? Just one option that can take
in a combination name or 'all' would be sufficient ?
##########
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##########
@@ -357,6 +357,34 @@ The Huge File tests validate Azure storages's ability to
handle large files —t
Tests at this scale are slow: they are best executed from hosts running in
the cloud infrastructure where the storage endpoint is based.
+##No test no review: Run different combinations of tests using the runtests.sh
script
Review comment:
Also move under "Testing Azure ABFS Client"
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,224 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+testresourcesdir=src/test/resources
+combconfsdir=$testresourcesdir/combconfs
+combtestfile=$testresourcesdir/abfs-combination-test-configs.xml
+
+shouldbuild=1
+
+logdir=dev-support/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+properties=
+values=
+
+validate() {
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$combination" ]; then
+ echo "Exiting. combination cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for combination $combination"
+ exit -1
+ fi
+
+ for filename in "${combinations[@]}"; do
+ if [[ ! -f "$combconfsdir/$filename.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ done
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+cleancombinationconfigs() {
+ rm -rf $combconfsdir
+ mkdir -p $combconfsdir
+}
+
+generateconfigs() {
+ combconffile="$combconfsdir/$combination.xml"
+ rm -rf "$combconffile"
+ cat > "$combconffile" << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $combination"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ formatxml "$combconffile"
+}
+
+formatxml() {
+ xmlstarlet fo -s 2 "$1" > "$1.tmp"
+ mv "$1.tmp" "$1"
+}
+
+setactiveconf() {
+ if [[ ! -f "$combconfsdir/$combination.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ rm -rf $combtestfile
+ cat > $combtestfile << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+ xmlstarlet ed -P -L -s /configuration -t elem -n include -v "" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n href -v
"combconfs/$combination.xml" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n xmlns -v
"http://www.w3.org/2001/XInclude"; $combtestfile
+ formatxml $combtestfile
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" "$combconffile"
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property "$combconffile"
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n
value -v "$2" "$combconffile"
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+summary() {
+ {
+ echo ""
+ echo "$combination"
+ echo "========================"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+ } >> "$testresultsfilename"
+ printf "\n----- Test results -----\n"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+
+ secondstaken=$((ENDTIME - STARTTIME))
+ mins=$((secondstaken / 60))
+ secs=$((secondstaken % 60))
+ printf "\nTime taken: %s mins %s secs.\n" "$mins" "$secs"
+ echo "Find test logs for the combination ($combination) in: $testlogfilename"
+ echo "Find consolidated test results in: $testresultsfilename"
+ echo "----------"
+}
+
+init() {
+ checkdependencies
+ if [[ "$shouldbuild" -eq "1" ]]; then
+ if ! mvn clean install -DskipTests
+ then
+ echo ""
+ echo "Exiting. Build failed."
+ exit -1
+ fi
+ fi
+ starttime=$(date +"%Y-%m-%d_%H-%M-%S")
+ mkdir -p "$logdir"
+ testresultsfilename="$logdir/$starttime/Test-Results.txt"
+ if [[ -z "$combinations" ]]; then
+ combinations=( $( ls $combconfsdir/*.xml ))
+ fi
+}
+
+runtests() {
+ parseoptions "$@"
+ validate
+ if [ -z "$starttime" ]; then
+ init
+ fi
+ shopt -s nullglob
+ for combconffile in "${combinations[@]}"; do
+ STARTTIME=$(date +%s)
+ combination=$(basename "$combconffile" .xml)
+ mkdir -p "$logdir/$starttime"
+ testlogfilename="$logdir/$starttime/Test-Logs-$combination.txt"
+ printf "\nRunning the combination: %s..." "$combination"
+ setactiveconf
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount
verify >> "$testlogfilename" || true
+ ENDTIME=$(date +%s)
+ summary
+ done
+}
+
+begin() {
+ cleancombinationconfigs
+}
+
+parseoptions() {
+ while getopts ":c:a:n?t:" option; do
+ case "${option}" in
+ a)
+ combination=$(basename "$OPTARG" .xml)
+ setactiveconf
+ exit 0
+ ;;
+ c)
+ combination=$(basename "$OPTARG" .xml)
+ combinations+=("$combination")
+ ;;
+ n)
+ shouldbuild=0
+ ;;
+ t)
+ threadcount=$OPTARG
+ ;;
+ *|?|h)
+ echo "Usage: $0 [-n] [-a COMBINATION_NAME] [-c COMBINATION_NAME] [-t
THREAD_COUNT]"
Review comment:
Camel case script names
##########
File path: hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh
##########
@@ -0,0 +1,224 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+testresourcesdir=src/test/resources
+combconfsdir=$testresourcesdir/combconfs
+combtestfile=$testresourcesdir/abfs-combination-test-configs.xml
+
+shouldbuild=1
+
+logdir=dev-support/testlogs
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+threadcount=
+defaultthreadcount=8
+
+properties=
+values=
+
+validate() {
+ if [ -z "$threadcount" ] ; then
+ threadcount=$defaultthreadcount
+ fi
+ numberegex='^[0-9]+$'
+ if ! [[ $threadcount =~ $numberegex ]] ; then
+ echo "Exiting. The script param (threadcount) should be a number"
+ exit -1
+ fi
+ if [ -z "$combination" ]; then
+ echo "Exiting. combination cannot be empty"
+ exit -1
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for combination $combination"
+ exit -1
+ fi
+
+ for filename in "${combinations[@]}"; do
+ if [[ ! -f "$combconfsdir/$filename.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ done
+}
+
+checkdependencies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+ echo "Exiting. pcregrep is required to run the script."
+ exit -1
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+ echo "Exiting. xmlstarlet is required to run the script."
+ exit -1
+ fi
+}
+
+cleancombinationconfigs() {
+ rm -rf $combconfsdir
+ mkdir -p $combconfsdir
+}
+
+generateconfigs() {
+ combconffile="$combconfsdir/$combination.xml"
+ rm -rf "$combconffile"
+ cat > "$combconffile" << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+ echo "Exiting. Number of properties and values differ for $combination"
+ exit -1
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+ key=${properties[$i]}
+ val=${values[$i]}
+ changeconf "$key" "$val"
+ done
+ formatxml "$combconffile"
+}
+
+formatxml() {
+ xmlstarlet fo -s 2 "$1" > "$1.tmp"
+ mv "$1.tmp" "$1"
+}
+
+setactiveconf() {
+ if [[ ! -f "$combconfsdir/$combination.xml" ]]; then
+ echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
+ exit -1
+ fi
+ rm -rf $combtestfile
+ cat > $combtestfile << ENDOFFILE
+<configuration>
+
+</configuration>
+ENDOFFILE
+ xmlstarlet ed -P -L -s /configuration -t elem -n include -v "" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n href -v
"combconfs/$combination.xml" $combtestfile
+ xmlstarlet ed -P -L -i /configuration/include -t attr -n xmlns -v
"http://www.w3.org/2001/XInclude"; $combtestfile
+ formatxml $combtestfile
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" "$combconffile"
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property "$combconffile"
+ if ! xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n
value -v "$2" "$combconffile"
+ then
+ echo "Exiting. Changing config property failed."
+ exit -1
+ fi
+}
+
+summary() {
+ {
+ echo ""
+ echo "$combination"
+ echo "========================"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+ } >> "$testresultsfilename"
+ printf "\n----- Test results -----\n"
+ pcregrep -M "$testresultsregex" "$testlogfilename"
+
+ secondstaken=$((ENDTIME - STARTTIME))
+ mins=$((secondstaken / 60))
+ secs=$((secondstaken % 60))
+ printf "\nTime taken: %s mins %s secs.\n" "$mins" "$secs"
+ echo "Find test logs for the combination ($combination) in: $testlogfilename"
+ echo "Find consolidated test results in: $testresultsfilename"
+ echo "----------"
+}
+
+init() {
+ checkdependencies
+ if [[ "$shouldbuild" -eq "1" ]]; then
+ if ! mvn clean install -DskipTests
+ then
+ echo ""
+ echo "Exiting. Build failed."
+ exit -1
+ fi
+ fi
+ starttime=$(date +"%Y-%m-%d_%H-%M-%S")
+ mkdir -p "$logdir"
+ testresultsfilename="$logdir/$starttime/Test-Results.txt"
+ if [[ -z "$combinations" ]]; then
+ combinations=( $( ls $combconfsdir/*.xml ))
+ fi
+}
+
+runtests() {
+ parseoptions "$@"
+ validate
+ if [ -z "$starttime" ]; then
+ init
+ fi
+ shopt -s nullglob
+ for combconffile in "${combinations[@]}"; do
+ STARTTIME=$(date +%s)
+ combination=$(basename "$combconffile" .xml)
+ mkdir -p "$logdir/$starttime"
+ testlogfilename="$logdir/$starttime/Test-Logs-$combination.txt"
+ printf "\nRunning the combination: %s..." "$combination"
+ setactiveconf
+ mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount
verify >> "$testlogfilename" || true
+ ENDTIME=$(date +%s)
+ summary
+ done
+}
+
+begin() {
+ cleancombinationconfigs
+}
+
+parseoptions() {
+ while getopts ":c:a:n?t:" option; do
+ case "${option}" in
+ a)
+ combination=$(basename "$OPTARG" .xml)
+ setactiveconf
+ exit 0
+ ;;
+ c)
+ combination=$(basename "$OPTARG" .xml)
+ combinations+=("$combination")
+ ;;
+ n)
+ shouldbuild=0
+ ;;
+ t)
+ threadcount=$OPTARG
+ ;;
+ *|?|h)
+ echo "Usage: $0 [-n] [-a COMBINATION_NAME] [-c COMBINATION_NAME] [-t
THREAD_COUNT]"
+ echo ""
+ echo "Where:"
+ echo " -a COMBINATION_NAME Specify the combination name which needs
to be activated."
+ echo " -c COMBINATION_NAME Specify the combination name for test
runs"
+ echo " -n Specify this option if there is no need to build before
running the tests"
Review comment:
Do not provide 'dont build' option. When each of the test combo
triggers, it will start with build and a failure will not be visible until the
test completes. Better to know at the start itself if build has a prob so that
the dev can stop further run of the script.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 506990)
Time Spent: 7h 50m (was: 7h 40m)
> ABFS: Run the integration tests with various combinations of configurations
> and publish a consolidated results
> --------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-17191
> URL: https://issues.apache.org/jira/browse/HADOOP-17191
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure, test
> Affects Versions: 3.3.0
> Reporter: Bilahari T H
> Assignee: Bilahari T H
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> ADLS Gen 2 supports accounts with and without hierarchical namespace support.
> ABFS driver supports various authorization mechanisms like OAuth, haredKey,
> Shared Access Signature. The integration tests need to be executed against
> accounts with and without hierarchical namespace support using various
> authorization mechanisms.
> Currently the developer has to manually run the tests with different
> combinations of configurations ex: HNS account with SharedKey and OAuth,
> NonHNS account with SharedKey etc..
> The expectation is to automate these runs with different combinations. This
> will help the developer to run the integration tests with different variants
> of configurations automatically.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
