Yuanhao Lu created OOZIE-3605:
---------------------------------
Summary: ShareLib installation does not honor dfs.replication in
HDFS configuration
Key: OOZIE-3605
URL: https://issues.apache.org/jira/browse/OOZIE-3605
Project: Oozie
Issue Type: Bug
Affects Versions: 5.2.0, 5.1.0
Reporter: Yuanhao Lu
The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
replication factor to be 3 when using multi-threaded copying while
single-threaded copy will follow the replication factor in hdfs-site.xml.
[https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
[https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
This could be problematic when a cluster has less than 3 data nodes. Since the
replication can never be 3 in this case, hdfs report will show a lot of files
are under replicated.
{code:java}
$ hdfs dfsadmin -report | head
Configured Capacity: 148067303424 (137.90 GB)
Present Capacity: 147612037120 (137.47 GB)
DFS Remaining: 145914187776 (135.89 GB)
DFS Used: 1697849344 (1.58 GB)
DFS Used%: 1.15%
Under replicated blocks: 1003
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
{code}
And the message from hdfs fsck will be like
{code:java}
/user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
and 0 decommissioning replica(s).
/user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
and 0 decommissioning replica(s).
/user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
and 0 decommissioning replica(s).
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
