[jira] [Created] (HDFS-15763) Provide NameNode Analytics as a NameNode plugin
Plamen Jeliazkov created HDFS-15763: --- Summary: Provide NameNode Analytics as a NameNode plugin Key: HDFS-15763 URL: https://issues.apache.org/jira/browse/HDFS-15763 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov https://github.com/paypal/NNAnalytics NameNode Analytics is a tool that has been used in production for several years now for maintaining the namespace of large HDFS instances by allowing better querying of an existing namespace via a specialized NameNode. This JIRA is to track bringing NameNode Analytics to the official HDFS project as a simple NameNode plugin, configurable for any Standby NameNode. More details will be included here in time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-14068) Allow manual transition from Standby to Observer
Plamen Jeliazkov created HDFS-14068: --- Summary: Allow manual transition from Standby to Observer Key: HDFS-14068 URL: https://issues.apache.org/jira/browse/HDFS-14068 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Plamen Jeliazkov With automatic failover enabled, I am unable to make use of the new transitionToObserver HAAdmin command. This JIRA is to remove the limitation and allow manual transition between Standby and Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-14068) Allow manual transition from Standby to Observer
[ https://issues.apache.org/jira/browse/HDFS-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov resolved HDFS-14068. - Resolution: Duplicate > Allow manual transition from Standby to Observer > > > Key: HDFS-14068 > URL: https://issues.apache.org/jira/browse/HDFS-14068 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Plamen Jeliazkov >Priority: Major > > With automatic failover enabled, I am unable to make use of the new > transitionToObserver HAAdmin command. This JIRA is to remove the limitation > and allow manual transition between Standby and Observer. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13665) Move RPC response serialization into Server.doResponse
Plamen Jeliazkov created HDFS-13665: --- Summary: Move RPC response serialization into Server.doResponse Key: HDFS-13665 URL: https://issues.apache.org/jira/browse/HDFS-13665 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov In HDFS-13399 we addressed a race condition in AlignmentContext processing where the RPC response would assign a transactionId independently of the transactions own processing, resulting in a stateId response that was lower than expected. See here: https://issues.apache.org/jira/browse/HDFS-13399?focusedCommentId=16464279=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16464279 And here: https://issues.apache.org/jira/browse/HDFS-13399?focusedCommentId=16498660=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16498660 At the end if was agreed upon to move the logic of Server.setupResponse into Server.doResponse directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13399) Make Client field AlignmentContext non-static.
Plamen Jeliazkov created HDFS-13399: --- Summary: Make Client field AlignmentContext non-static. Key: HDFS-13399 URL: https://issues.apache.org/jira/browse/HDFS-13399 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-12943 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov In HDFS-12977, DFSClient's constructor was altered to make use of a new static method in Client that allowed one to set an AlignmentContext. This work is to remove that static field and make each DFSClient pass it's AlignmentContext down to the proxy Call level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
Plamen Jeliazkov created HDFS-13331: --- Summary: Add lastSeenStateId to RpcRequestHeader. Key: HDFS-13331 URL: https://issues.apache.org/jira/browse/HDFS-13331 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-12977 added a stateId into the RpcResponseHeader which is returned by NameNode and stored by DFSClient. This JIRA is to followup on that work and have the DFSClient send their stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can then compare with their own and act accordingly. This JIRA work focuses on just the part of making DFSClient send their state through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-13020) Add JMX metrics for computeReconstruction and computeInvalidation work
Plamen Jeliazkov created HDFS-13020: --- Summary: Add JMX metrics for computeReconstruction and computeInvalidation work Key: HDFS-13020 URL: https://issues.apache.org/jira/browse/HDFS-13020 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor HDFS configuration allows tweaking of NameNode replication settings. However, optimal settings may be tricky to determine and usually rely on observation of key metrics in order to determine. In one particular instance, I was trying to find the optimal value of 'dfs.namenode.replication.work.multiplier.per.iteration' but was unable to find any metrics around the ReconstructionMonitor's replication and invalidation cycles that is mostly affected by changing that property. If we expose some trivial JMX metrics for how long these cycles take we can better understand what effects tweaking the work multiplier will have. Of course, I am also open to suggestions around what else we should track here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Reopened] (HDFS-7906) BlockReport of a RUR can have a FINALIZED original replica from Truncate
[ https://issues.apache.org/jira/browse/HDFS-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov reopened HDFS-7906: BlockReport of a RUR can have a FINALIZED original replica from Truncate Key: HDFS-7906 URL: https://issues.apache.org/jira/browse/HDFS-7906 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-7886 revealed that there is a new replica state change as part of Truncate. It is now possible that a FINALIZED replica become a RUR replica with an original replica pointing back to the FINALIZED replica. As a result, it is possible to hit an assert in BlockLIstAsLongs.setBlock() whereby you are trying to add a FINALIZED replica to the ucList because a RUR only had an under-construction original replica prior to Truncate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7906) BlockReport of a RUR can have a FINALIZED original replica from Truncate
[ https://issues.apache.org/jira/browse/HDFS-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov resolved HDFS-7906. Resolution: Duplicate BlockReport of a RUR can have a FINALIZED original replica from Truncate Key: HDFS-7906 URL: https://issues.apache.org/jira/browse/HDFS-7906 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-7886 revealed that there is a new replica state change as part of Truncate. It is now possible that a FINALIZED replica become a RUR replica with an original replica pointing back to the FINALIZED replica. As a result, it is possible to hit an assert in BlockLIstAsLongs.setBlock() whereby you are trying to add a FINALIZED replica to the ucList because a RUR only had an under-construction original replica prior to Truncate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7906) BlockReport of a RUR can have a FINALIZED original replica from Truncate
[ https://issues.apache.org/jira/browse/HDFS-7906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov resolved HDFS-7906. Resolution: Won't Fix BlockReport of a RUR can have a FINALIZED original replica from Truncate Key: HDFS-7906 URL: https://issues.apache.org/jira/browse/HDFS-7906 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-7886 revealed that there is a new replica state change as part of Truncate. It is now possible that a FINALIZED replica become a RUR replica with an original replica pointing back to the FINALIZED replica. As a result, it is possible to hit an assert in BlockLIstAsLongs.setBlock() whereby you are trying to add a FINALIZED replica to the ucList because a RUR only had an under-construction original replica prior to Truncate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7906) BlockReport of a RUR can have a FINALIZED original replica from Truncate
Plamen Jeliazkov created HDFS-7906: -- Summary: BlockReport of a RUR can have a FINALIZED original replica from Truncate Key: HDFS-7906 URL: https://issues.apache.org/jira/browse/HDFS-7906 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.7.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-7886 revealed that there is a new replica state change as part of Truncate. It is now possible that a FINALIZED replica become a RUR replica with an original replica pointing back to the FINALIZED replica. As a result, it is possible to hit an assert in BlockLIstAsLongs.setBlock() whereby you are trying to add a FINALIZED replica to the ucList because a RUR only had an under-construction replica prior to Truncate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7315) DFSTestUtil.readFileBuffer opens extra FSDataInputStream
Plamen Jeliazkov created HDFS-7315: -- Summary: DFSTestUtil.readFileBuffer opens extra FSDataInputStream Key: HDFS-7315 URL: https://issues.apache.org/jira/browse/HDFS-7315 Project: Hadoop HDFS Issue Type: Bug Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Trivial DFSTestUtil.readFileBuffer() calls FileSystem.open() twice. Once just under the try statement, and once inside the IOUtils.copyBytes() call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-5675) Add Mkdirs operation to NNThroughputBenchmark
Plamen Jeliazkov created HDFS-5675: -- Summary: Add Mkdirs operation to NNThroughputBenchmark Key: HDFS-5675 URL: https://issues.apache.org/jira/browse/HDFS-5675 Project: Hadoop HDFS Issue Type: Bug Components: benchmarks Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 3.0.0 I did some work to extend NNThroughputBenchmark that I would like to contribute to the community. It is pretty straightforward; just adding a Mkdir operation to the test in order to see the operations per second of a multiple 'mkdir' commands. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-4559) WebHDFS does not allow resolution of Symlinks
Plamen Jeliazkov created HDFS-4559: -- Summary: WebHDFS does not allow resolution of Symlinks Key: HDFS-4559 URL: https://issues.apache.org/jira/browse/HDFS-4559 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Fix For: 3.0.0, 2.0.4-beta WebHDFS allows you to create symlinks via the CREATESYMLINK operation, but the GETFILEINFO operation specifically calls the getFileInfo() method of the NameNodeRpcServer which does not resolve symlinks. I propose adding a parameter to GETFILEINFO such that if true will call getFileLinkInfo() rather than getFileInfo() which will resolve any symlinks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4561) Symlinks broken in DFS
Plamen Jeliazkov created HDFS-4561: -- Summary: Symlinks broken in DFS Key: HDFS-4561 URL: https://issues.apache.org/jira/browse/HDFS-4561 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Fix For: 3.0.0, 2.0.4-beta Symlinks appear to be working when using FileContext, but broken when using DFS / NameNodeRpcServer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4475) OutOfMemory by BPService.offerService() takes down DataNode
Plamen Jeliazkov created HDFS-4475: -- Summary: OutOfMemory by BPService.offerService() takes down DataNode Key: HDFS-4475 URL: https://issues.apache.org/jira/browse/HDFS-4475 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Fix For: 3.0.0, 2.0.3-alpha In DataNode, there are catchs around BPService.offerService() call but no catch for OutOfMemory as there is for the DataXeiver as introduced in 0.22.0. The issue can be replicated like this: 1) Create a cluster of X DataNodes and 1 NameNode and low memory settings (-Xmx128M or something similar). 2) Flood HDFS with of file creation. 3) DataNodes will hit OoM, stop blockpool service, and shutdown. The resolution is to catch the OoMException and handle it properly when calling BlockPool.offerService() in DataNode.java; like as done in 0.22.0 of Hadoop. DataNodes should not shutdown or crash but remain in a sort of frozen state until memory issues are resolved by GC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4422) Upgrade servlet-api dependency from version 2.5 to 3.0.
Plamen Jeliazkov created HDFS-4422: -- Summary: Upgrade servlet-api dependency from version 2.5 to 3.0. Key: HDFS-4422 URL: https://issues.apache.org/jira/browse/HDFS-4422 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Minor Fix For: 2.0.3-alpha Please update the servlet-api jar from 2.5 to javax.servlet 3.0 via Maven: dependency groupIdjavax.servlet/groupId artifactIdjavax.servlet-api/artifactId version3.0.1/version scopeprovided/scope /dependency I am running a 2.0.3 dev-cluster and can confirm compatibility. I have removed the servlet-api-2.5.jar file and replaced it with javax.servlet-3.0.jar file. I am using javax.servlet-3.0 because it implements methods that I use for a filter, namely the HttpServletResponse.getStatus() method. I believe it is a gain to have this dependency as it allows more functionality and has so far proven to be backwards compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2189) guava-r09 dependency missing from ivy/hadoop-hdfs-template.xml in HDFS.
[ https://issues.apache.org/jira/browse/HDFS-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov reopened HDFS-2189: This patch needs to be reverted. After doing a new build we noticed this patch did not resolve anything. The real fix lies within MAPREDUCE's template XML file; will be resolved later. guava-r09 dependency missing from ivy/hadoop-hdfs-template.xml in HDFS. - Key: HDFS-2189 URL: https://issues.apache.org/jira/browse/HDFS-2189 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Priority: Blocker Fix For: 0.22.0 Attachments: patch.txt Corrected version of: https://issues.apache.org/jira/browse/MAPREDUCE-2627 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2189) guava-r09 dependency missing from ivy/hadoop-hdfs-template.xml in HDFS.
guava-r09 dependency missing from ivy/hadoop-hdfs-template.xml in HDFS. - Key: HDFS-2189 URL: https://issues.apache.org/jira/browse/HDFS-2189 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Reporter: Plamen Jeliazkov Priority: Blocker Fix For: 0.22.0 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2151) restartDFS causing a hang when running on Windows. Makes other test cases fail.
restartDFS causing a hang when running on Windows. Makes other test cases fail. --- Key: HDFS-2151 URL: https://issues.apache.org/jira/browse/HDFS-2151 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Environment: Windows 7, Eclipse Reporter: Plamen Jeliazkov When running all the namenode testcases on Windows via Eclipse, testRestartDFS runs first, however after printing out Shutting down the Mini HDFS Cluster, it proceeds to hang, leading to timeout, and causing other tests to get an error saying Cannot remove directory X. This may strictly be a Windows environment error. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2151) restartDFS causing a hang when running on Windows. Makes other test cases fail.
[ https://issues.apache.org/jira/browse/HDFS-2151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov resolved HDFS-2151. Resolution: Fixed Fix Version/s: 0.22.0 Error was in the core-site.xml file. Needed to comment out the location property so it would not try to access existing directories. restartDFS causing a hang when running on Windows. Makes other test cases fail. --- Key: HDFS-2151 URL: https://issues.apache.org/jira/browse/HDFS-2151 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Environment: Windows 7, Eclipse Reporter: Plamen Jeliazkov Labels: hdfs, testRestartDFS, tests Fix For: 0.22.0 Original Estimate: 168h Remaining Estimate: 168h When running all the namenode testcases on Windows via Eclipse, testRestartDFS runs first, however after printing out Shutting down the Mini HDFS Cluster, it proceeds to hang, leading to timeout, and causing other tests to get an error saying Cannot remove directory X. Not to mention the test itself takes over 10 minutes to run. This may strictly be a Windows environment error. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
