adoroszlai opened a new pull request #1537: HDDS-2187. ozone-mr test fails with No FileSystem for scheme o3fs URL: https://github.com/apache/hadoop/pull/1537 ## What changes were proposed in this pull request? MR acceptance tests are [failing](https://github.com/elek/ozone-ci/blob/2f2c99652af6b26a95f08eece9e545f0d72ccf45/trunk/trunk-nightly-20190925-htsvn/acceptance/output.log#L724) at the `Create user dir for hadoop` step of `Create directories required for MR test` suite. This happens since [HDDS-2101](https://issues.apache.org/jira/browse/HDDS-2101) removed `fs.o3fs.impl` definition from `core-site.xml`, filesystem provider is defined in META-INF of `hadoop-ozone-filesystem-lib-*.jar`. The problem is that the filesystem jars are not on the classpath for `ozone fs` commands used to create hadoop user dir in the `ozone-mr` env, so the explicit config is needed. This change adds back the config, but only for scm, not for Hadoop. It also applies both changes (this fix and the removal per HDDS-2101) to `ozonesecure-mr` env. It is based on top of #1533 (which should be merged first). https://issues.apache.org/jira/browse/HDDS-2187 ## How was this patch tested? Ran all affected acceptance tests locally. ``` 0 datanode is up and healthy (until now) 3 datanodes are up and registered to the scm ============================================================================== hadoop32-createmrenv :: Create directories required for MR test ============================================================================== Create test volume, bucket and key | PASS | ------------------------------------------------------------------------------ Create user dir for hadoop | PASS | ------------------------------------------------------------------------------ hadoop32-createmrenv :: Create directories required for MR test | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-createmrenv-scm.xml ============================================================================== hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs ============================================================================== Test hadoop dfs | PASS | ------------------------------------------------------------------------------ hadoop32-hadoopo3fs :: Test ozone fs with hadoopfs | PASS | 1 critical test, 1 passed, 0 failed 1 test total, 1 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-hadoopo3fs-rm.xml ============================================================================== hadoop32-mapreduce :: Execute MR jobs ============================================================================== Execute PI calculation | PASS | ------------------------------------------------------------------------------ Execute WordCount | PASS | ------------------------------------------------------------------------------ hadoop32-mapreduce :: Execute MR jobs | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop32/result/robot-hadoop32-hadoop32-mapreduce-rm.xml Robot framework is not installed, the reports can be generated (sudo pip install robotframework). 0 datanode is up and healthy (until now) 3 datanodes are up and registered to the scm ============================================================================== hadoop31-createmrenv :: Create directories required for MR test ============================================================================== Create test volume, bucket and key | PASS | ------------------------------------------------------------------------------ Create user dir for hadoop | PASS | ------------------------------------------------------------------------------ hadoop31-createmrenv :: Create directories required for MR test | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-createmrenv-scm.xml fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/7) Installing libbz2 (1.0.6-r7) (2/7) Installing libffi (3.2.1-r4) (3/7) Installing gdbm (1.13-r1) (4/7) Installing sqlite-libs (3.25.3-r1) (5/7) Installing python2 (2.7.15-r2) (6/7) Installing py-setuptools (39.1.0-r0) (7/7) Installing py2-pip (10.0.1-r0) Executing busybox-1.28.4-r0.trigger Executing glibc-bin-2.27-r0.trigger OK: 93 MiB in 45 packages Collecting robotframework Downloading https://files.pythonhosted.org/packages/22/0f/1b9ffa0c4e59789b50e6034866e823b7d4a5c7eaedad7bfd0bba42f2aa9d/robotframework-3.1.2-py2.py3-none-any.whl (602kB) Installing collected packages: robotframework Successfully installed robotframework-3.1.2 ============================================================================== hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs ============================================================================== Test hadoop dfs | PASS | ------------------------------------------------------------------------------ hadoop31-hadoopo3fs :: Test ozone fs with hadoopfs | PASS | 1 critical test, 1 passed, 0 failed 1 test total, 1 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-hadoopo3fs-rm.xml ============================================================================== hadoop31-mapreduce :: Execute MR jobs ============================================================================== Execute PI calculation | PASS | ------------------------------------------------------------------------------ Execute WordCount | PASS | ------------------------------------------------------------------------------ hadoop31-mapreduce :: Execute MR jobs | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop31/result/robot-hadoop31-hadoop31-mapreduce-rm.xml Robot framework is not installed, the reports can be generated (sudo pip install robotframework). 2 datanode is up and healthy (until now) 3 datanodes are up and registered to the scm ============================================================================== hadoop27-createmrenv :: Create directories required for MR test ============================================================================== Create test volume, bucket and key | PASS | ------------------------------------------------------------------------------ Create user dir for hadoop | PASS | ------------------------------------------------------------------------------ hadoop27-createmrenv :: Create directories required for MR test | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-createmrenv-scm.xml fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz (1/7) Installing libbz2 (1.0.6-r7) (2/7) Installing libffi (3.2.1-r4) (3/7) Installing gdbm (1.13-r1) (4/7) Installing sqlite-libs (3.25.3-r1) (5/7) Installing python2 (2.7.15-r2) (6/7) Installing py-setuptools (39.1.0-r0) (7/7) Installing py2-pip (10.0.1-r0) Executing busybox-1.28.4-r0.trigger Executing glibc-bin-2.27-r0.trigger OK: 93 MiB in 45 packages Collecting robotframework Downloading https://files.pythonhosted.org/packages/22/0f/1b9ffa0c4e59789b50e6034866e823b7d4a5c7eaedad7bfd0bba42f2aa9d/robotframework-3.1.2-py2.py3-none-any.whl (602kB) Installing collected packages: robotframework Successfully installed robotframework-3.1.2 ============================================================================== hadoop27-hadoopo3fs :: Test ozone fs with hadoopfs ============================================================================== Test hadoop dfs | PASS | ------------------------------------------------------------------------------ hadoop27-hadoopo3fs :: Test ozone fs with hadoopfs | PASS | 1 critical test, 1 passed, 0 failed 1 test total, 1 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-hadoopo3fs-rm.xml ============================================================================== hadoop27-mapreduce :: Execute MR jobs ============================================================================== Execute PI calculation | PASS | ------------------------------------------------------------------------------ Execute WordCount | PASS | ------------------------------------------------------------------------------ hadoop27-mapreduce :: Execute MR jobs | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/hadoop27/result/robot-hadoop27-hadoop27-mapreduce-rm.xml Robot framework is not installed, the reports can be generated (sudo pip install robotframework). 0 datanode is up and healthy (until now) 0 datanode is up and healthy (until now) 0 datanode is up and healthy (until now) 3 datanodes are up and registered to the scm ============================================================================== ozonesecure-mr-kinit :: Kinit test user ============================================================================== Kinit | PASS | ------------------------------------------------------------------------------ ozonesecure-mr-kinit :: Kinit test user | PASS | 1 critical test, 1 passed, 0 failed 1 test total, 1 passed, 0 failed ============================================================================== Output: /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-kinit-om.xml ============================================================================== ozonesecure-mr-createmrenv :: Create directories required for MR test ============================================================================== Create test volume, bucket and key | PASS | ------------------------------------------------------------------------------ Create user dir for hadoop | PASS | ------------------------------------------------------------------------------ ozonesecure-mr-createmrenv :: Create directories required for MR test | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-createmrenv-om.xml ============================================================================== ozonesecure-mr-kinit-hadoop :: Kinit test user ============================================================================== Kinit | PASS | ------------------------------------------------------------------------------ ozonesecure-mr-kinit-hadoop :: Kinit test user | PASS | 1 critical test, 1 passed, 0 failed 1 test total, 1 passed, 0 failed ============================================================================== Output: /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-kinit-hadoop-rm.xml ============================================================================== ozonesecure-mr-mapreduce :: Execute MR jobs ============================================================================== Execute PI calculation | PASS | ------------------------------------------------------------------------------ Execute WordCount | PASS | ------------------------------------------------------------------------------ ozonesecure-mr-mapreduce :: Execute MR jobs | PASS | 2 critical tests, 2 passed, 0 failed 2 tests total, 2 passed, 0 failed ============================================================================== Output: /tmp/smoketest/ozonesecure-mr/result/robot-ozonesecure-mr-ozonesecure-mr-mapreduce-rm.xml Robot framework is not installed, the reports can be generated (sudo pip install robotframework). ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
