[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186832#comment-17186832
]
Hadoop QA commented on OOZIE-3605:
--
Testing JIRA OOZIE-3605
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [15] new bugs found below threshold in total
that must be fixed.
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [core].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:red}-1{color} There are [15] new bugs found below threshold in
[tools] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
.At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
.At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
.At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
.At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
.{color:green}+1{color} There are no new bugs found in [server].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [webapp].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3215
.{color:orange}Tests failed at first run:{color}
TestBlockingInputStream#testLimitedWritingBlockingInputStream
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://ci-hadoop.apache.org/job/PreCommit-OOZIE-Build/5/
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/a
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186812#comment-17186812
]
Gézapeti commented on OOZIE-3605:
-
:sigh: I'm sorry. I've messed up the commit message - again :(
I can't fix it unfortunately as we don't do force pushes. The release log file
is good and that's more importatn
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186805#comment-17186805
]
Yuanhao Lu commented on OOZIE-3605:
---
[~gezapeti]
Thanks for taking a look. Seems there is a mismatch for commit message (and
also for the previous one)?
https://github.com/apache/oozie/commit/7bcd9819dd825dc906f03978cd4e967c75d108b0
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Fix For: trunk
>
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186800#comment-17186800
]
Gézapeti commented on OOZIE-3605:
-
This is a nice catch!
Thanks for the contribution [~luyuanhao]!
+1, committed to master!
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17186797#comment-17186797
]
Hadoop QA commented on OOZIE-3605:
--
PreCommit-OOZIE-Build started
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17173589#comment-17173589
]
Yuanhao Lu commented on OOZIE-3605:
---
The "[15] new bugs found" seems not relevant to the change. Could anyone please
take a look?
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166053#comment-17166053
]
Hadoop QA commented on OOZIE-3605:
--
Testing JIRA OOZIE-3605
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any star imports
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} Javadoc generation succeeded with the patch
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warning(s)
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:red}-1{color} There are [15] new bugs found below threshold in total
that must be fixed.
.{color:green}+1{color} There are no new bugs found in [webapp].
.{color:green}+1{color} There are no new bugs found in
[fluent-job/fluent-job-api].
.{color:green}+1{color} There are no new bugs found in [sharelib/streaming].
.{color:green}+1{color} There are no new bugs found in [sharelib/spark].
.{color:green}+1{color} There are no new bugs found in [sharelib/sqoop].
.{color:green}+1{color} There are no new bugs found in [sharelib/hcatalog].
.{color:green}+1{color} There are no new bugs found in [sharelib/git].
.{color:green}+1{color} There are no new bugs found in [sharelib/distcp].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive].
.{color:green}+1{color} There are no new bugs found in [sharelib/pig].
.{color:green}+1{color} There are no new bugs found in [sharelib/hive2].
.{color:green}+1{color} There are no new bugs found in [sharelib/oozie].
.{color:red}-1{color} There are [15] new bugs found below threshold in
[tools] that must be fixed, listing only the first [5] ones.
.You can find the SpotBugs diff here (look for the red and orange ones):
tools/findbugs-new.html
.The top [5] most important SpotBugs errors are:
.At OozieDBCLI.java:[line 584]: This use of
java/sql/Statement.executeUpdate(Ljava/lang/String;)I can be vulnerable to SQL
injection
.At OozieDBCLI.java:[line 574]: At OozieDBCLI.java:[line 573]
.At OozieDBCLI.java:[line 577]: At OozieDBCLI.java:[line 575]
.At OozieDBCLI.java:[line 579]: At OozieDBCLI.java:[line 578]
.At OozieDBCLI.java:[line 584]: At OozieDBCLI.java:[line 581]
.{color:orange}0{color} There are [4] new bugs found in [server] that would
be nice to have fixed.
.You can find the SpotBugs diff here: server/findbugs-new.html
.{color:green}+1{color} There are no new bugs found in [examples].
.{color:green}+1{color} There are no new bugs found in [docs].
.{color:green}+1{color} There are no new bugs found in [client].
.{color:green}+1{color} There are no new bugs found in [core].
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:green}+1 TESTS{color}
.Tests run: 3215
.{color:orange}Tests failed at first run:{color}
TestBlockingInputStream#testFastWritingBlockingInputStream
TestBlockingInputStream#testLimitedWritingBlockingInputStream
.For the complete list of flaky tests, see TEST-SUMMARY-FULL files.
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:green}+1 MODERNIZER{color}
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/PreCommit-OOZIE-Build/1322/
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using
[jira] [Commented] (OOZIE-3605) ShareLib installation does not honor dfs.replication in HDFS configuration
[
https://issues.apache.org/jira/browse/OOZIE-3605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166019#comment-17166019
]
Hadoop QA commented on OOZIE-3605:
--
PreCommit-OOZIE-Build started
> ShareLib installation does not honor dfs.replication in HDFS configuration
> --
>
> Key: OOZIE-3605
> URL: https://issues.apache.org/jira/browse/OOZIE-3605
> Project: Oozie
> Issue Type: Bug
>Affects Versions: 5.1.0, 5.2.0
>Reporter: Yuanhao Lu
>Priority: Major
> Attachments: OOZIE-3605-001.patch
>
>
> The change in https://issues.apache.org/jira/browse/OOZIE-2791 hardcoded
> replication factor to be 3 when using multi-threaded copying while
> single-threaded copy will follow the replication factor in hdfs-site.xml.
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L391]
> [https://github.com/apache/oozie/blob/master/tools/src/main/java/org/apache/oozie/tools/OozieSharelibCLI.java#L306]
> This could be problematic when a cluster has less than 3 data nodes. Since
> the replication can never be 3 in this case, hdfs report will show a lot of
> files are under replicated.
> {code:java}
> $ hdfs dfsadmin -report | head
> Configured Capacity: 148067303424 (137.90 GB)
> Present Capacity: 147612037120 (137.47 GB)
> DFS Remaining: 145914187776 (135.89 GB)
> DFS Used: 1697849344 (1.58 GB)
> DFS Used%: 1.15%
> Under replicated blocks: 1003
> Blocks with corrupt replicas: 0
> Missing blocks: 0
> Missing blocks (with replication factor 1): 0
> Pending deletion blocks: 0
> {code}
> And the message from hdfs fsck will be like
> {code:java}
> /user/oozie/share/lib/lib_20200707223334/git/commons-codec-1.10.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742826_2002.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/commons-lang3-3.3.2.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742810_1986.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> /user/oozie/share/lib/lib_20200707223334/git/httpclient-4.5.9.jar: Under
> replicated BP-1985902824-10.65.207.110-1594161186186:blk_1073742815_1991.
> Target Replicas is 3 but found 2 live replica(s), 0 decommissioned replica(s)
> and 0 decommissioning replica(s).
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
