[jira] [Comment Edited] (HDFS-15333) Vulnerability fixes need for jackson-databinding HDFS dependency library
[ https://issues.apache.org/jira/browse/HDFS-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153209#comment-17153209 ] weiyanen edited comment on HDFS-15333 at 7/8/20, 3:40 AM: -- So NOW, how can I resolve this vulnerability problem? I've used htrace-core4-4.1.0-incubating and it used jackson 2.4.0 which has vulnerability issues. I must use htrace-core4-4.1.0-incubating, otherwise, I would get an error for "org/apache/htrace/core/Tracer$Builder Context: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder". Can we just ignore the Vulnerability Issue although code scan throw out this issue? Because "No JSON deserialization is involved the code path. Even JSON serialization is only used in specific span receivers which is barely used." was (Author: weiyanen): So NOW, how can I resolve this vulnerability problem? I've used htrace-core4-4.1.0-incubating and it used jackson 2.4.0 which has vulnerability issues. I must use htrace-core4-4.1.0-incubating, otherwise, I would get an error for "org/apache/htrace/core/Tracer$Builder Context: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder". > Vulnerability fixes need for jackson-databinding HDFS dependency library > > > Key: HDFS-15333 > URL: https://issues.apache.org/jira/browse/HDFS-15333 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.2.1 > Environment: [^hdfs_imagescan_result.csv] >Reporter: Hridesh >Priority: Critical > Attachments: hdfs_imagescan_result.csv > > > HDFS has couple of dependency which is having jackson library with > vulnerability. > Below are list of library used by HDFS which is having vulnerability: > * htrace-core4-4.1.0-incubating.jar:jackson-databind > * htrace-core-3.1.0-incubating.jar:jackson-databind > * aws-java-sdk-bundle-1.11.375.jar:jackson-databind > * hadoop-client-runtime-3.2.1.jar:jackson-databind > * jackson-databind-2.9.8.jar > * hadoop-client-runtime-3.2.1.jar:jackson-databind > > For example: "htrace-core4-4.1.0-incubating" build with jackson 2.4.0. POM > URL: > [https://github.com/apache/incubator-retired-htrace/blob/e12b5fcfaafa56d676fee5f873da01df6b61dac9/pom.xml.] > > Jackson version < 2.9.1 has below list of vulnerabilities: > CVE-2019-14379 > CVE-2019-16335 > CVE-2019-17531 > CVE-2019-14540 > CVE-2018-11307 > CVE-2019-12402 > CVE-2018-7489 > CVE-2018-12022 > CVE-2019-14439 > CVE-2017-15095 > CVE-2017-7525 > CVE-2017-17485 > > Attaching image scan result file. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15333) Vulnerability fixes need for jackson-databinding HDFS dependency library
[ https://issues.apache.org/jira/browse/HDFS-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153209#comment-17153209 ] weiyanen commented on HDFS-15333: - So NOW, how can I resolve this vulnerability problem? I've used htrace-core4-4.1.0-incubating and it used jackson 2.4.0 which has vulnerability issues. I must use htrace-core4-4.1.0-incubating, otherwise, I would get an error for "org/apache/htrace/core/Tracer$Builder Context: java.lang.NoClassDefFoundError: org/apache/htrace/core/Tracer$Builder". > Vulnerability fixes need for jackson-databinding HDFS dependency library > > > Key: HDFS-15333 > URL: https://issues.apache.org/jira/browse/HDFS-15333 > Project: Hadoop HDFS > Issue Type: Improvement > Components: security >Affects Versions: 3.2.1 > Environment: [^hdfs_imagescan_result.csv] >Reporter: Hridesh >Priority: Critical > Attachments: hdfs_imagescan_result.csv > > > HDFS has couple of dependency which is having jackson library with > vulnerability. > Below are list of library used by HDFS which is having vulnerability: > * htrace-core4-4.1.0-incubating.jar:jackson-databind > * htrace-core-3.1.0-incubating.jar:jackson-databind > * aws-java-sdk-bundle-1.11.375.jar:jackson-databind > * hadoop-client-runtime-3.2.1.jar:jackson-databind > * jackson-databind-2.9.8.jar > * hadoop-client-runtime-3.2.1.jar:jackson-databind > > For example: "htrace-core4-4.1.0-incubating" build with jackson 2.4.0. POM > URL: > [https://github.com/apache/incubator-retired-htrace/blob/e12b5fcfaafa56d676fee5f873da01df6b61dac9/pom.xml.] > > Jackson version < 2.9.1 has below list of vulnerabilities: > CVE-2019-14379 > CVE-2019-16335 > CVE-2019-17531 > CVE-2019-14540 > CVE-2018-11307 > CVE-2019-12402 > CVE-2018-7489 > CVE-2018-12022 > CVE-2019-14439 > CVE-2017-15095 > CVE-2017-7525 > CVE-2017-17485 > > Attaching image scan result file. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15448) When starting a DataNode, call BlockPoolManager#startAll() twice.
[ https://issues.apache.org/jira/browse/HDFS-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153188#comment-17153188 ] jianghua zhu commented on HDFS-15448: - [~elgoiri] , your suggestions are very helpful to me. Here I resubmitted a new patch file (HDFS-15448.003.patch). [~elgoiri] , [~hexiaoqiao] , can you help review again? > When starting a DataNode, call BlockPoolManager#startAll() twice. > - > > Key: HDFS-15448 > URL: https://issues.apache.org/jira/browse/HDFS-15448 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.1 >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Attachments: HDFS-15448.001.patch, HDFS-15448.002.patch, > HDFS-15448.003.patch, method_invoke_path.jpg > > > When starting a DataNode, call BlockPoolManager#startAll() twice. > The first call: > BlockPoolManager#doRefreshNamenodes() > private void doRefreshNamenodes( > Map> addrMap, > Map> lifelineAddrMap) > throws IOException { > ... > startAll(); > ... > } > The second call: > DataNode#runDatanodeDaemon() > public void runDatanodeDaemon() throws IOException { > blockPoolManager.startAll(); > ... > } -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15448) When starting a DataNode, call BlockPoolManager#startAll() twice.
[ https://issues.apache.org/jira/browse/HDFS-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jianghua zhu updated HDFS-15448: Attachment: HDFS-15448.003.patch > When starting a DataNode, call BlockPoolManager#startAll() twice. > - > > Key: HDFS-15448 > URL: https://issues.apache.org/jira/browse/HDFS-15448 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.1 >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Attachments: HDFS-15448.001.patch, HDFS-15448.002.patch, > HDFS-15448.003.patch, method_invoke_path.jpg > > > When starting a DataNode, call BlockPoolManager#startAll() twice. > The first call: > BlockPoolManager#doRefreshNamenodes() > private void doRefreshNamenodes( > Map> addrMap, > Map> lifelineAddrMap) > throws IOException { > ... > startAll(); > ... > } > The second call: > DataNode#runDatanodeDaemon() > public void runDatanodeDaemon() throws IOException { > blockPoolManager.startAll(); > ... > } -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15455) Expose HighestPriorityReplBlocks and LastReplicaBlocks statistics
[ https://issues.apache.org/jira/browse/HDFS-15455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152973#comment-17152973 ] Íñigo Goiri commented on HDFS-15455: Thanks [~surmountian], can you post a backport of HDFS-13658 for branch-2.9? > Expose HighestPriorityReplBlocks and LastReplicaBlocks statistics > - > > Key: HDFS-15455 > URL: https://issues.apache.org/jira/browse/HDFS-15455 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.9.0, 2.9.2 >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > > Similar to HDFS-13658, blocks with only 1 replica may cause the lost of > customer data, so we need to make this awared and take action if necessary. > This change is for HDFS 2.9 as we will still be using it for some time and > switching to HDFS 3.X is not simple work. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15448) When starting a DataNode, call BlockPoolManager#startAll() twice.
[ https://issues.apache.org/jira/browse/HDFS-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152972#comment-17152972 ] Íñigo Goiri commented on HDFS-15448: [~hexiaoqiao] already went through the code paths. We may want to add a check to createDataNode() and runDatanodeDaemon() to verify that the block manager is started. I would make sure that the tests for createDataNode() checks for something like testBPServiceState(). > When starting a DataNode, call BlockPoolManager#startAll() twice. > - > > Key: HDFS-15448 > URL: https://issues.apache.org/jira/browse/HDFS-15448 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.1 >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Attachments: HDFS-15448.001.patch, HDFS-15448.002.patch, > method_invoke_path.jpg > > > When starting a DataNode, call BlockPoolManager#startAll() twice. > The first call: > BlockPoolManager#doRefreshNamenodes() > private void doRefreshNamenodes( > Map> addrMap, > Map> lifelineAddrMap) > throws IOException { > ... > startAll(); > ... > } > The second call: > DataNode#runDatanodeDaemon() > public void runDatanodeDaemon() throws IOException { > blockPoolManager.startAll(); > ... > } -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15425) Review Logging of DFSClient
[ https://issues.apache.org/jira/browse/HDFS-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152868#comment-17152868 ] Hudson commented on HDFS-15425: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18417 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18417/]) HDFS-15425. Review Logging of DFSClient. Contributed by Hongbing Wang. (hexiaoqiao: rev 4f26454a7d1b560f959cdb2fb0641147a85642da) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripeReader.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedInputStream.java > Review Logging of DFSClient > --- > > Key: HDFS-15425 > URL: https://issues.apache.org/jira/browse/HDFS-15425 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient >Reporter: Hongbing Wang >Assignee: Hongbing Wang >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15425.001.patch, HDFS-15425.002.patch, > HDFS-15425.003.patch > > > Review use of SLF4J for DFSClient.LOG. > Make the code more concise and readable. > Less is more ! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15425) Review Logging of DFSClient
[ https://issues.apache.org/jira/browse/HDFS-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15425: --- Component/s: dfsclient > Review Logging of DFSClient > --- > > Key: HDFS-15425 > URL: https://issues.apache.org/jira/browse/HDFS-15425 > Project: Hadoop HDFS > Issue Type: Improvement > Components: dfsclient >Reporter: Hongbing Wang >Assignee: Hongbing Wang >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15425.001.patch, HDFS-15425.002.patch, > HDFS-15425.003.patch > > > Review use of SLF4J for DFSClient.LOG. > Make the code more concise and readable. > Less is more ! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15425) Review Logging of DFSClient
[ https://issues.apache.org/jira/browse/HDFS-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He updated HDFS-15425: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) commit to trunk. Thanks [~wanghongbing] for your contributions! Thanks [~elgoiri] for reviews! > Review Logging of DFSClient > --- > > Key: HDFS-15425 > URL: https://issues.apache.org/jira/browse/HDFS-15425 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Hongbing Wang >Assignee: Hongbing Wang >Priority: Minor > Fix For: 3.4.0 > > Attachments: HDFS-15425.001.patch, HDFS-15425.002.patch, > HDFS-15425.003.patch > > > Review use of SLF4J for DFSClient.LOG. > Make the code more concise and readable. > Less is more ! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152729#comment-17152729 ] Hadoop QA commented on HDFS-15025: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 26m 7s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:blue}0{color} | {color:blue} prototool {color} | {color:blue} 0m 0s{color} | {color:blue} prototool was not available. {color} | | {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m 0s{color} | {color:blue} markdownlint was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 16 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 22m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 11s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 49s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 46s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 18m 46s{color} | {color:red} root generated 26 new + 136 unchanged - 26 fixed = 162 total (was 162) {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 46s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 3m 5s{color} | {color:orange} root: The patch generated 23 new + 726 unchanged - 3 fixed = 749 total (was 729) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 39s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 28s{color} | {color:red} hadoop-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 39s{color} | {color:red} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} |
[jira] [Updated] (HDFS-15025) Applying NVDIMM storage media to HDFS
[ https://issues.apache.org/jira/browse/HDFS-15025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hadoop_hdfs_hw updated HDFS-15025: -- Attachment: HDFS-15025.001.patch Status: Patch Available (was: Open) > Applying NVDIMM storage media to HDFS > - > > Key: HDFS-15025 > URL: https://issues.apache.org/jira/browse/HDFS-15025 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, hdfs >Reporter: hadoop_hdfs_hw >Priority: Major > Attachments: Applying NVDIMM to HDFS.pdf, HDFS-15025.001.patch, > NVDIMM_patch(WIP).patch > > > The non-volatile memory NVDIMM is faster than SSD, it can be used > simultaneously with RAM, DISK, SSD. The data of HDFS stored directly on > NVDIMM can not only improves the response rate of HDFS, but also ensure the > reliability of the data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15448) When starting a DataNode, call BlockPoolManager#startAll() twice.
[ https://issues.apache.org/jira/browse/HDFS-15448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152556#comment-17152556 ] jianghua zhu commented on HDFS-15448: - [~elgoiri] , [~linyiqun] , can you give me some more suggestions on this issue? Thanks a lot. > When starting a DataNode, call BlockPoolManager#startAll() twice. > - > > Key: HDFS-15448 > URL: https://issues.apache.org/jira/browse/HDFS-15448 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.1 >Reporter: jianghua zhu >Assignee: jianghua zhu >Priority: Major > Attachments: HDFS-15448.001.patch, HDFS-15448.002.patch, > method_invoke_path.jpg > > > When starting a DataNode, call BlockPoolManager#startAll() twice. > The first call: > BlockPoolManager#doRefreshNamenodes() > private void doRefreshNamenodes( > Map> addrMap, > Map> lifelineAddrMap) > throws IOException { > ... > startAll(); > ... > } > The second call: > DataNode#runDatanodeDaemon() > public void runDatanodeDaemon() throws IOException { > blockPoolManager.startAll(); > ... > } -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-15455) Expose HighestPriorityReplBlocks and LastReplicaBlocks statistics
Xiao Liang created HDFS-15455: - Summary: Expose HighestPriorityReplBlocks and LastReplicaBlocks statistics Key: HDFS-15455 URL: https://issues.apache.org/jira/browse/HDFS-15455 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.9.2, 2.9.0 Reporter: Xiao Liang Assignee: Xiao Liang Similar to HDFS-13658, blocks with only 1 replica may cause the lost of customer data, so we need to make this awared and take action if necessary. This change is for HDFS 2.9 as we will still be using it for some time and switching to HDFS 3.X is not simple work. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org