[
https://issues.apache.org/jira/browse/HADOOP-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614026#comment-17614026
]
ASF GitHub Bot commented on HADOOP-17767:
-----------------------------------------
steveloughran commented on code in PR #3124:
URL: https://github.com/apache/hadoop/pull/3124#discussion_r989967669
##########
hadoop-tools/hadoop-azure/.gitignore:
##########
@@ -1,5 +1,6 @@
.checkstyle
bin/
-src/test/resources/combinationConfigFiles
src/test/resources/abfs-combination-test-configs.xml
dev-support/testlogs
+src/test/resources/accountSettings/*
+!src/test/resources/accountSettings/accountName_settings.xml.template
Review Comment:
can you add a newline
##########
hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh:
##########
@@ -15,117 +15,88 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-testresourcesdir=src/test/resources
-combconfsdir=$testresourcesdir/combinationConfigFiles
-combtestfile=$testresourcesdir/abfs-combination-test-configs.xml
+resourceDir=src/test/resources/
+accountSettingsFolderName=accountSettings
+combtestfile=$resourceDir
+combtestfile+=abfs-combination-test-configs.xml
+logdir=dev-support/testlogs/
-logdir=dev-support/testlogs
testresultsregex="Results:(\n|.)*?Tests run:"
-testresultsfilename=
-starttime=
-threadcount=
-defaultthreadcount=8
-
-properties=
-values=
-
-validate() {
- if [ -z "$threadcount" ] ; then
- threadcount=$defaultthreadcount
- fi
- numberegex='^[0-9]+$'
- if ! [[ $threadcount =~ $numberegex ]] ; then
- echo "Exiting. The script param (threadcount) should be a number"
- exit -1
- fi
- if [ -z "$combination" ]; then
- echo "Exiting. combination cannot be empty"
- exit -1
- fi
- propertiessize=${#properties[@]}
- valuessize=${#values[@]}
- if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
- echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for combination $combination"
- exit -1
- fi
-
- for filename in "${combinations[@]}"; do
- if [[ ! -f "$combconfsdir/$filename.xml" ]]; then
- echo "Exiting. Combination config file ($combconfsdir/$combination.xml)
does not exist."
- exit -1
- fi
- done
-}
-
-checkdependencies() {
- if ! [ "$(command -v pcregrep)" ]; then
- echo "Exiting. pcregrep is required to run the script."
- exit -1
- fi
- if ! [ "$(command -v xmlstarlet)" ]; then
- echo "Exiting. xmlstarlet is required to run the script."
- exit -1
+accountConfigFileSuffix="_settings.xml"
+testOutputLogFolder=$logdir
+testlogfilename=combinationTestLogFile
+
+fullRunStartTime=$(date +%s)
+STARTTIME=$(date +%s)
+ENDTIME=$(date +%s)
+
+outputFormatOn="\033[0;95m"
+outputFormatOff="\033[0m"
+
+triggerRun()
+{
+ echo ' '
+ combination=$1
+ accountName=$2
+ runTest=$3
+ processcount=$4
+ cleanUpTestContainers=$5
+
+ if [ -z "$accountName" ]; then
+ logOutput "ERROR: Test account not configured. Re-run the script and
choose SET_OR_CHANGE_TEST_ACCOUNT to configure the test account."
+ exit 0;
Review Comment:
exit with an error code
##########
hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md:
##########
@@ -602,26 +602,64 @@ various test combinations, it will:
2. Run tests for all combinations
3. Summarize results across all the test combination runs.
-As a pre-requisite step, fill config values for test accounts and credentials
-needed for authentication in `src/test/resources/azure-auth-keys.xml.template`
-and rename as `src/test/resources/azure-auth-keys.xml`.
+Below are the pre-requiste steps to follow:
+1. Copy
-**To add a new test combination:** Templates for mandatory test combinations
-for PR validation are present in `dev-support/testrun-scripts/runtests.sh`.
-If a new one needs to be added, add a combination set within
-`dev-support/testrun-scripts/runtests.sh` similar to the ones already defined
-and
-1. Provide a new combination name
-2. Update properties and values array which need to be effective for the test
-combination
-3. Call generateconfigs
+ ./src/test/resources/azure-auth-keys.xml.template
+ TO
+ ./src/test/resources/azure-auth-keys.xml
+ Update account names that should be used in the test run for HNS and non-HNS
+ combinations in the 2 properties present in the xml (account name should be
+ without domain part), namely
+
+ fs.azure.hnsTestAccountName
+ fs.azure.nonHnsTestAccountName
+ azure-auth-keys.xml is listed in .gitignore, so any accidental account name
leak is prevented.
Review Comment:
add below
```
XInclude is supported, so for extra security secrets may be
kept out of the source tree then referenced through an an XInclude element:
<include xmlns="http://www.w3.org/2001/XInclude";
href="/users/self/.secrets/auth-keys.xml" />
```
As this is what I do everyone else should do the same
##########
hadoop-tools/hadoop-azure/dev-support/testrun-scripts/testsupport.sh:
##########
@@ -134,108 +105,68 @@ summary() {
echo "$combination"
echo "========================"
pcregrep -M "$testresultsregex" "$testlogfilename"
- } >> "$testresultsfilename"
+ } >> "$aggregatedTestResult"
printf "\n---
> ABFS: Improve test scripts
> --------------------------
>
> Key: HADOOP-17767
> URL: https://issues.apache.org/jira/browse/HADOOP-17767
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.3.0
> Reporter: Sneha Vijayarajan
> Assignee: Sneha Vijayarajan
> Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Current test run scripts need manual update across all combinations in
> runTests.sh for account name and is working off a single azure-auth-keys.xml
> file. While having to test across accounts that span various geo, the config
> file grows big and also needs a manual change for configs such as
> fs.contract.test.[abfs/abfss] which has to be uniquely set. To use the script
> across various combinations, dev to be aware of the names of all the
> combinations defined in runTests.sh as well.
>
> These concerns are addressed in the new version of the scripts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
