[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r503193523 ## File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md ## @@ -80,7 +80,7 @@ tests against mocked storage, which is an in-memory emulation of Azure Storage. The integration tests are designed to test directly against an Azure storage service, and require an account and credentials in order to run. -This is done by creating the file to `src/test/resources/azure-auth-keys.xml` +This is done by creating the file to `src/test/resources/abfs-testrun-configs.xml` Review comment: Basically the script runs the following command mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=$threadcount verify >> "$testlogfilename" So this is intended for ABFS This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r503192587 ## File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template ## @@ -0,0 +1,181 @@ + + + + + + +
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r500506248
##
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please
use the following:
{IP}:{PORT}
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
+the auth configs for all the accounts used for testing are provided in
+abfs-testrun-configs.xml. In case any new flags or properties are introduced
+with the code change, add the combinations with the possible configurations
+into the runtests.sh. The thread count can be specified as the command line
+argument for the script. By default the same will be 8.
+
+Adding a combination of tests involves setting the variable scenario (ex: HNS
+-OAuth) and specifying the specific configurations for the particular
+combination with 2 arrays namely properties and values. Specify the property
+names within the array properties and corresponding values in the values
+array. The property and value is determined by the array index. The value for
+the property mentioned at index 1 of array properties should be specified at
+index 1 of the array values. Call the function runtestwithconfs once the 3
+values mentioned are set. Now the script runtests.sh is ready to be ran.
+
+Once the tests are completed, logs will be present in the directory
+target/testlogs. A consolidated test results will be present in the file
+Test-$starttime-Results.log, $startname will be the start time of the test.
+Similarly, the full test report can be found in individual log files, for each
+of the scenarios with the file name Test-$starttime-Logs-$scenario. Please
+attach the consolidates test results from the file Test-$starttime-Results.log
Review comment:
Done
##
File path: hadoop-tools/hadoop-azure/src/site/markdown/testing_azure.md
##
@@ -935,3 +935,30 @@ http[s]://[account][domain-suffix]/[filesystem], please
use the following:
{IP}:{PORT}
```
+
+##Run different combinations of tests using the runtests.sh script
+
+This is the expected way in which the tests have to be ran before raising a PR.
+The script runtests.sh contain template for 3 combinations of tests. Ensure
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r486299349 ## File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template ## @@ -0,0 +1,156 @@ + + + + + + +
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r485828996 ## File path: hadoop-tools/hadoop-azure/.gitignore ## @@ -1,2 +1,4 @@ .checkstyle -bin/ \ No newline at end of file +bin/ +testlogs Review comment: Done This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485828835
##
File path: hadoop-tools/hadoop-azure/testsupport.sh
##
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for scenario $scenario"
+exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+echo "Exiting. pcregrep is required to run the script."
+exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+echo "Exiting. xmlstarlet is required to run the script."
+exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value
-v "$2" $conffile
Review comment:
Could not find the same documented with the xmlstarlet documentation or
produce anything returning non zero. But still added a check for the safer side.
##
File path: hadoop-tools/hadoop-azure/testsupport.sh
##
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for scenario $scenario"
+exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+echo "Exiting. pcregrep is required to run the script."
+exit
+ fi
+ if ! [ "$(command -v xmlstarlet)" ]; then
+echo "Exiting. xmlstarlet is required to run the script."
+exit
+ fi
+}
+
+changeconf() {
+ xmlstarlet ed -P -L -d "/configuration/property[name='$1']" $conffile
+ xmlstarlet ed -P -L -s /configuration -t elem -n propertyTMP -v "" -s
/configuration/propertyTMP -t elem -n name -v "$1" -r
/configuration/propertyTMP -v property $conffile
+ xmlstarlet ed -P -L -s "/configuration/property[name='$1']" -t elem -n value
-v "$2" $conffile
+}
+
+testwithconfs() {
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -ne "$valuessize" ]; then
+echo "Exiting. Number of properties and values differ for $scenario"
+exit
+ fi
+ for ((i = 0; i < propertiessize; i++)); do
+key=${properties[$i]}
+val=${values[$i]}
+
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485827879
##
File path: hadoop-tools/hadoop-azure/testsupport.sh
##
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for scenario $scenario"
+exit
+ fi
+}
+
+checkdependancies() {
+ if ! [ "$(command -v pcregrep)" ]; then
+echo "Exiting. pcregrep is required to run the script."
+exit
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485827246
##
File path: hadoop-tools/hadoop-azure/testsupport.sh
##
@@ -0,0 +1,109 @@
+#!/usr/bin/env bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+conffile=src/test/resources/abfs-testrun-configs.xml
+bkpconffile=src/test/resources/abfs-testrun-configs_BKP.xml
+testresultsregex="Results:(\n|.)*?Tests run:"
+testresultsfilename=
+starttime=
+
+validate() {
+ if [ -z "$scenario" ]; then
+ echo "Exiting. scenario cannot be empty"
+ exit
+ fi
+ propertiessize=${#properties[@]}
+ valuessize=${#values[@]}
+ if [ "$propertiessize" -lt 1 ] || [ "$valuessize" -lt 1 ] || [
"$propertiessize" -ne "$valuessize" ]; then
+echo "Exiting. Both properties and values arrays has to be populated and
of same size. Please check for scenario $scenario"
+exit
+ fi
+}
+
+checkdependancies() {
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r485823482 ## File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template ## @@ -0,0 +1,156 @@ + + + + + + + + + + +fs.azure.account.auth.type +SharedKey + + + +
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278: URL: https://github.com/apache/hadoop/pull/2278#discussion_r485818694 ## File path: hadoop-tools/hadoop-azure/src/test/resources/abfs-testrun-configs.xml.template ## @@ -0,0 +1,156 @@ + + + + + + +
[GitHub] [hadoop] bilaharith commented on a change in pull request #2278: HADOOP-17191. ABFS: Run the integration tests with various combinations of configurations and publish a consolidated results
bilaharith commented on a change in pull request #2278:
URL: https://github.com/apache/hadoop/pull/2278#discussion_r485816830
##
File path:
hadoop-tools/hadoop-azure/src/test/java/org/apache/hadoop/fs/azurebfs/ITestAzureBlobFileSystemDelegationSAS.java
##
@@ -64,7 +64,7 @@
LoggerFactory.getLogger(ITestAzureBlobFileSystemDelegationSAS.class);
public ITestAzureBlobFileSystemDelegationSAS() throws Exception {
-// These tests rely on specific settings in azure-auth-keys.xml:
+// These tests rely on specific settings in abfs-testrun-configs.xml:
Review comment:
Basically we are renaming azure-auth-keys to abfs-testrun-configs.
Because we expect the same to contain non auth related configs too.
In the main config file (azure-test.xml the xinclude is now made to
abfs-testrun-configs)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
