[
https://issues.apache.org/jira/browse/HDDS-1525?focusedWorklogId=274027&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-274027
]
ASF GitHub Bot logged work on HDDS-1525:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Jul/19 12:45
Start Date: 09/Jul/19 12:45
Worklog Time Spent: 10m
Work Description: elek commented on issue #1065: HDDS-1525. Mapreduce
failure when using Hadoop 2.7.5
URL: https://github.com/apache/hadoop/pull/1065#issuecomment-509627399
As it's a bigger patch here is a quick guidance what is included:
```
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/rpc/RpcClient.java
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzFs.java
```
This is the original fix by @xiaoyuyao It makes the RpcClient compatible
with older hadoop versions
````
hadoop-ozone/dist/src/main/compose/ozone-mr
hadoop-ozone/dist/src/main/compose/ozone-fs
```
ozone-mr dir is reorganized to have subdirectories for mapreduce tests
(hadoop 2.7, 3.1, 3.2).
Smoketest shell scripts are added
ozone-fs dir is deleted as ozone-mr includes all the functionality and it's
better structured.
```
hadoop-ozone/dist/src/main/compose/testlib.sh
```
Improved to pass custom robot test arguments. you case see the usage in
ozone-mr/hadoop27/test.sh
```
hadoop-ozone/dist/src/main/smoketest/ozonefs/hadoopo3fs.robot
```
Trival fix, the test was wrong until now.
```
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneClientAdapterImpl.java
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/FileStatusAdapter.java
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/FilteredClassLoader.java
hadoop-ozone/ozonefs/src/main/java/org/apache/hadoop/fs/ozone/OzoneClientAdapter.java
```
This is the second part of the fix (on top of the fixes from @xiaoyuyao. As
the FileStatus (hadoop3) is not compatible with FileStatus (hadoop2), we need a
class which can be shared between hadoop3 and hadoop2 (FileStatusAdapter,
includes only simple java types).
```
hadoop-ozone/ozonefs-lib-legacy/pom.xml
```
This is the third part of the fix. The token files should be available for
hadoop2 classloader even if they are the cluster is not secure (to
serialize/unserialize tokens). The required classes are copied directly to the
jar file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 274027)
Time Spent: 20m (was: 10m)
> Mapreduce failure when using Hadoop 2.7.5
> -----------------------------------------
>
> Key: HDDS-1525
> URL: https://issues.apache.org/jira/browse/HDDS-1525
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Filesystem
> Affects Versions: 0.4.0
> Reporter: Sammi Chen
> Assignee: Elek, Marton
> Priority: Blocker
> Labels: pull-request-available
> Attachments: HDDS-1525.poc.patch, teragen.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Integrate Ozone(0.4 branch) with Hadoop 2.7.5, "hdfs dfs -ls /" can pass,
> while teragen failed.
> When add -verbose:class to java options, it shows that class KeyProvider is
> loaded twice by different classloader while it is only loaded once when
> execute "hdfs dfs -ls /"
> All jars under share/ozone/lib are added into hadoop classpath except ozone
> file system current lib jar.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
