[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764197#comment-16764197 ] Surendra Singh Lilhore commented on HDFS-10636: --- Thanks [~virajith], created new HDFS-14263 to fix this. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch, > HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch, > HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch, > HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762972#comment-16762972 ] Virajith Jalaparti commented on HDFS-10636: --- [~surendrasingh] agree. I think I added this as a check but it doesn't need to be there. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch, > HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch, > HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch, > HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762681#comment-16762681 ] Surendra Singh Lilhore commented on HDFS-10636: --- [~virajith], I have one doubt for this patch. Why do we need block file exists check for {{LocalReplica}} while opening input stream for block ? {code:java} + if(info != null && info.blockDataExists()) { + return info.getDataInputStream(seekOffset); } else { - try { - return openAndSeek(blockFile, seekOffset); - } catch (FileNotFoundException fnfe) { {code} I got {{requestShortCircuitFds()}} response little slower than 2.7.2. I checked the code and found one extra {{File.exists()}} check added here. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch, > HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch, > HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch, > HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638560#comment-16638560 ] Íñigo Goiri commented on HDFS-10636: I don't think Yetus for branch-2.9 works. This is the patch we recently backported internally so +1. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti >Priority: Major > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch, > HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch, > HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch, > HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827020#comment-15827020 ] Andrew Wang commented on HDFS-10636: Thanks [~virajith]! [~eddyxu] could you add a release note to this JIRA explaining the potential incompatibility? > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826954#comment-15826954 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~andrew.wang], Yes, you are right in that all changes are to the Private APIs in the datanode. However, [~eddyxu] mentioned that some vendors might have implemented solutions based on {{FinalizedReplica}} (see [this|https://issues.apache.org/jira/browse/HDFS-10636?focusedCommentId=15429061=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15429061] comment). I think this was the reason the JIRA was marked incompatible. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819714#comment-15819714 ] Andrew Wang commented on HDFS-10636: Question, is this JIRA actually incompatible? I did a quick skim, and it only seems to change things in the DN that are unannotated (thus defacto Private) or annotated Private. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488318#comment-15488318 ] Virajith Jalaparti commented on HDFS-10636: --- Great! Thanks [~eddyxu]. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488302#comment-15488302 ] Hudson commented on HDFS-10636: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10438 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10438/]) HDFS-10636. Modify ReplicaInfo to remove the assumption that replica (lei: rev 86c9862bec0248d671e657aa56094a2919b8ac14) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeFaultInjector.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/TestExternalDataset.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTransferRbw.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FinalizedReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockPoolSliceStorage.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * (edit)
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487931#comment-15487931 ] Lei (Eddy) Xu commented on HDFS-10636: -- Hi, [~virajith] Thanks for the insights. bq. Our solution is to have a ProvidedReplica class which inherits ReplicaInfo, and then have ProvidedFinalizedReplica inherit ProvidedReplica. It makes sense now. +1. Will commit shortly > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486024#comment-15486024 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~eddyxu], That is a very valid concern. Our solution is to have a {{ProvidedReplica}} class which inherits {{ReplicaInfo}}, and then have {{ProvidedFinalizedReplica}} inherit {{ProvidedReplica}}. Not only this, we will need to have a hierarchy for {{ProvidedReplica}} that mirrors the hierarchy for {{LocalReplica}} in the patch. This is somewhat forced on us as different classes are used to implicitly implement the replica state machine (for example, {{ReplicaInfo::getBytesOnDisk()}} must resolve correctly in {{FsDatasetImpl}}). Thus, we will also need to have {{ProvidedReplicaInPipeline}}, {{ProvidedReplicaUnderRecovery}} and others which inherit {{ProvidedReplica}}. One way to avoid implementing such shadow classes for {{ProvidedReplica}} would be to explicitly implement a state machine for the replicas. This would be a much more disruptive change, with significant amount of new code and changes required to several parts of {{FsDatasetImpl}}, and {{FsVolumeImpl}}. The current patch keeps most of the code paths intact with relatively minor changes to the APIs of {{ReplicaInfo}}. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485831#comment-15485831 ] Lei (Eddy) Xu commented on HDFS-10636: -- Sorry for late reply, [~chris] and [~virajith] I think that my previous thoughts on types might put too much constraints and make things difficult. Apologize for that. This patch itself looks OK to me. +0. for the reason described below: {code} abstract public class LocalReplica extends ReplicaInfo { } // and public class FinalizedReplica extends LocalReplica { } {code} My concern is that in the future, where in the class hierarchy is {{ProvidedFinalizedReplica}}, should it inherent from {{FinalizedReplica}} or {{LocalReplica}}? In such cases, is {{ProvidedFinalizedReplica}} a {{LocalReplica}}? > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485773#comment-15485773 ] Chris Douglas commented on HDFS-10636: -- The latest version of the patch lgtm, +1. Thanks [~virajith] for seeing this through. [~eddyxu] do you have any other feedback on v10 before commit? > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478899#comment-15478899 ] Virajith Jalaparti commented on HDFS-10636: --- The failed test case seems like a spurious failure (it runs fine on my dev machine). The patch does not affect that code path. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch, > HDFS-10636.009.patch, HDFS-10636.010.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478666#comment-15478666 ] Hadoop QA commented on HDFS-10636: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 24s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 15 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 23s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestRollingUpgrade | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10636 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827840/HDFS-10636.010.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux c5c352b23e5c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cba973f | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16694/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16694/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16694/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. >
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475575#comment-15475575 ] Hadoop QA commented on HDFS-10636: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 15 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10636 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827685/HDFS-10636.009.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 7fa7e1d0fd5b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 011f3b2 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt | | compile | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt | | javac | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | mvnsite | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt | | unit |
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472303#comment-15472303 ] Hadoop QA commented on HDFS-10636: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 15 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 39s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}101m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | | | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10636 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12827463/HDFS-10636.008.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2b335f745e06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d355573 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16663/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16663/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/16663/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/16663/console | | Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469060#comment-15469060 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~eddyxu], bq. Should {{LocalReplica}} and {{ProvidedReplica}} extend {{FinalizedReplica}}? Could you take a look at my previous [comment | https://issues.apache.org/jira/browse/HDFS-10636?focusedCommentId=15433727=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15433727] about this? I had a few questions as to what would be the motivation for this. bq. {{ReplicaBuilder#buildLocalReplicaInPipeline}} can be private. This will not be sufficient as the function is used in {{FsVolumeImpl}}. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468790#comment-15468790 ] Lei (Eddy) Xu commented on HDFS-10636: -- Hi, [~virajith], Thanks for the updates. And thanks for [~jpallas]'s inputs. I have a few questions remained: * Should {{LocalReplica}} and {{ProvidedReplica}} extend {{FinalizedReplica}}? It seems the opposite way in the current patch. Similarly concerns for {{ReplicaBeingWritten}} and {{LocalReplicaBeingWritten}}. * {{ReplicaBuilder#buildLocalReplicaInPipeline}} can be private. The rest looks good to me. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch, > HDFS-10636.006.patch, HDFS-10636.007.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468517#comment-15468517 ] Hadoop QA commented on HDFS-10636: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 15 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 25 new + 856 unchanged - 94 fixed = 881 total (was 950) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 43s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 94m 26s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDNFencing | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Issue | HDFS-10636 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12826989/HDFS-10636.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 3e2ad5721046 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f6c0b75 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/whitespace-eol.txt | | whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/whitespace-tabs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results |
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433904#comment-15433904 ] Joe Pallas commented on HDFS-10636: --- Possibly the only requirement to subclass {{FinalizedReplica}} today is that it appears in the dataset SPI, so any alternative implementation has to use it. The proposal removes that, so maybe there isn't a strong reason to have it be a type (if I understand the distinction I think [~virajith] is making between class as type and class as state). There are several places in the current patch where {{FinalizedReplica}} becomes {{ReplicaInfo}}, and I think that's a reasonable change. The concern [~eddyxu] expressed above about states and types is valid, but, in this case, I think the type system isn't really helping the programmer very much. Existing code using {{FinalizedReplica}} is already frequently stuck with only {{ReplicaInfo}} known at compile time, so a state check is needed anyway, and using a cast to {{FinalizedReplica}} after the state check doesn't add much. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433727#comment-15433727 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~eddyxu], bq. I know vendors who have already implemented similar solutions based on {{FinalizedReplica}}. I am trying to understand the use case for {{FinalizedReplica}} you mentioned. What properties of {{FinalizedReplica}} do you think we should preserve for these implementations to continue working? This question is also related to the following. bq. Have you considered about making {{LocalReplica}} and {{ProvidedReplica}} being subclasses of {{FinalizedReplica}}? {{FinalizedReplica}}, as implemented today, does not have any particular characteristics other than it's state and type. There do not exist functions in {{FinalizedReplica}} that are not in {{ReplicaInfo}}. Is the goal of your proposal to have {{FinalizedReplica}} used as a type, and not just state? Do you see having to do this for other states as well (RUR, RBW, RWR, TEMPORARY)? or having it for {{FinalizedReplica}} is more important? One concrete proposal for this can be the following: {{FinalizedReplica}} will be an abstract subclass of {{ReplicaInfo}} (overrides {{ReplicaInfo::getState()}}). We can then define {{LocalFinalizedReplica}} and {{ProvidedFinalizedReplica}}, which can be subclasses of {{FinalizedReplica}}. However, this would cause {{LocalFinalizedReplica}} to re-implement the functionality of other {{LocalReplica}}s. One way to avoid such re-implementation could be to have {{FinalizedReplica}} as an interface, with a {{FinalizedReplica::getReplicaInfo()}} function to perform all the {{ReplicaInfo}} related functions. Then {{LocalFinalizedReplica}} can subclass {{LocalReplica}} and implement the {{FinalizedReplica}} interface. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429061#comment-15429061 ] Lei (Eddy) Xu commented on HDFS-10636: -- bq. If not, are there existing cases where vendors might be using FinalizedReplica for non-File backed data? Yea. I know vendors who have already implemented similar solutions based on {{FinalizedReplica}}, because {{ProvidedReplica}} did not exist back to the days. But I think it might be OK for them if this patch did not break the their code. Another option might be that, "Finalized" looks like a state in the pipeline state machine, instead of a "Location" where the replica is stored, IMO. Have you considered about making {{LocalReplica}} and {{ProvidedReplica}} being subclasses of {{FinalizedReplica}}? So that for the pipeline state machine part of code, all the existing code can re-use the same {{FinalizedReplica}} as what it is today. Would love to hear your thoughts about it. bq. Having a good test coverage seems a cleaner way to solve this. Agree. That would be a nice thing to do. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427479#comment-15427479 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~eddyxu], bq. E.g., If you implement recovery logic in Azure or S3, the input should not be a File based replica. Good point. I agree with this. Having {{StorageLocation}} instead of {{File}} makes sense (other parts of the patch aim to already do this). I will make the necessary changes. bq. Would that still be the case for Azure? For HDFS-9806, the idea would be to have a {{ProvidedReplica}} which will be used to refer to data in external storages. As shown in HDFS-10675, this class would implement {{ReplicaInfo}} and {{ReplicaInPipeline}}, so that we can re-use the existing code for building replication pipelines and go through the block life cycle. bq. so I guess many vendors might expect using {{FinalizedReplica}} to directly represent the block data. Are you suggesting this in the context of HDFS-9806? If so, I think they should be using {{ProvidedReplica}} (as mentioned in my previous point). If not, are there existing cases where vendors might be using {{FinalizedReplica}} for non-{{File}} backed data? One way to deal with this issue with {{FinalizedReplica}} and the usage of class types is to have abstract classes for each replica type, and then have implementations for local and provided replicas. This will end up having a lot of classes which are essentially {{ReplicaInfo}}. Having a good test coverage seems a cleaner way to solve this. Thoughts? > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427349#comment-15427349 ] Lei (Eddy) Xu commented on HDFS-10636: -- Hi, [~virajith] Thanks a lot for update the patch and address the comments. And appreciate much for taking our comments into consideration. bq. The other way to deal with this would be to add setDir to ReplicaInfo but to avoid File as a parameter. What I am worried about is that, this constraint might not hold true for wider audience. E.g., If you implement recovery logic in Azure or S3, the input should not be a File based replica. bq.it should take in an URI. Which do you think is better? Maybe using {{StorageLocation}} as input? There was some efforts to abstract {{File}} to {{StorageLocation}}. bq. {{FinalizedReplica}} in the current type hierarchy, is a LocalReplica. Would that still be the case for Azure? I have saw a few cases that it is not a file-based solution. Also, {{FinalizedReplica}} should be the most common class in the {{ReplicaInfo}} hierarchy, so I guess many vendors might expect using {{FinalizedReplica}} to directly represent the block data. bq. The way I was thinking about it was that the state of ReplicaInfo should be known using ReplicaInfo::getState(), and not using the type system. I don't have strong option against the solution proposed here. The concerns of getting rid of type system and using {int} / {enum} to indicate the class type might make the other developer difficult to have good confident to write correct code, because like dynamic language as python/ruby, the bugs become hard to spot during compile time. If we can come up a good test coverage, the concern can be overcome to some extend. Thanks. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419293#comment-15419293 ] Virajith Jalaparti commented on HDFS-10636: --- Hi [~eddyxu], Thank you for the comments. I agree with points 1 and 2, and will fix them. bq. {{breakHardlinksIfNeeded}}, {{copyMetadata}} and {{copyBlockdata}} should not be in {{ReplicaInfo}}. Or should not use {{File}} as input. Agree that {{copyMetadata}} and {{copyBlockdata}} should not have {{File}} as a parameter. I will change it to {{URI}} to be more general. {{breakHardLinksIfNeeded}} has always been in {{ReplicaInfo}}. I made it abstract and moved the implementation to {{LocalReplica}}. bq. {{ReplicaUnderRecovery}}. Is there a way to avoid casting {{ReplicaInfo}} to {{LocalReplica}}? The only place where the fact that {{original}} is a {{LocalReplica}} matters is in {{ReplicaUnderRecovery::setDir()}}. One way to address this would be to add the cast only when {{original.setDir()}} is called. The other way to deal with this would be to add {{setDir}} to {{ReplicaInfo}} but to avoid {{File}} as a parameter, it should take in an URI. Which do you think is better? bq. In general, there are places in this patch that return {{ReplicaInfo}} for {{FinalizedReplica}}. which would makes type system weaker and is not future-proof. Is it necessary to be changed? This was intentional. The way I was thinking about it was that the state of {{ReplicaInfo}} should be known using {{ReplicaInfo::getState()}}, and not using the type system. The current code does the latter -- it uses the type system to ensure that replicas are in a certain state. Not relying on the type system and using the former (use {{ReplicaInfo::getState()}}) seems a cleaner way of doing this. What do you think? Also, {{FinalizedReplica}} in the current type hierarchy, is a {{LocalReplica}}. So, referring to replicas using {{FinalizedReplica}} assumes that they are {{LocalReplica}} and hence, backed by {{File}} s. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418219#comment-15418219 ] Lei (Eddy) Xu commented on HDFS-10636: -- Hi, [~virajith] Thanks for working on this. A few questions. * In {{FsDatasetImpl#fetchReplication}} , please remove the usage {{LOG.info}}. * In {{ReplicaInfo.java}}. The following comments seem irrelevant to the code below. {code} /** directory where block & meta files belong. */ {code} Also please do not remove the comments for the constroctors. * {{breakHardlinksIfNeeded}}, {{ copyMetadata}} and {{copyBlockdata}} should not be in {{ReplicaInfo}}. Or should not use {{File}} as input. * {{ReplicaUnderRecovery}}. Is there a way to avoid casting {{ReplicaInfo}} to {{LocalReplica}}? * In general, there are places in this patch that return {{ReplicaInfo}} for {{FinalizedReplica}}. which would makes type system weaker and is not future-proof. Is it necessary to be changed? Thanks. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408705#comment-15408705 ] Virajith Jalaparti commented on HDFS-10636: --- Thanks [~chris.douglas] for the comments and fixing the checkstyle/findbugs. I am attaching a new patch to address the above comments. bq. Breaking up the switch statement in build() avoids a checkstyle complaint? This might be more legible if each (fairly long) case stmt were in a private method. Addressed in the patch -- moved each case statement to a private function. bq. Is {{DataStorage::getTrashDirectoryForBlockFile}} now unused, except by tests? Yes, it is no longer used/required. I modified the tests that access {{DataStorage::getTrashDirectoryForBlockFile}}, to ensure that it is no longer needed and removed this function. I also modified tests that access {{BlockPoolSliceStorage::getTrashDirectory(File)}} to access {{BlockPoolSliceStorage::getTrashDirectory(ReplicaInfo)}}, and made the former a private function in {{BlockPoolSliceStorage}}. bq. the call to {{Storage::nativeCopyFileUnbuffered}} is removed, so there's no path to the {{NativeIO}} libraries. This should be maintained. Agree. I have restored this by adding two new abstract functions to {{ReplicaInfo}}: {{copyMetadata}} and {{copyBlockdata}}, and implementing them in {{LocalReplica}} to use {{Storage::nativeCopyFileUnbuffered}}. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti >Assignee: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch, > HDFS-10636.003.patch, HDFS-10636.004.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383319#comment-15383319 ] Virajith Jalaparti commented on HDFS-10636: --- Uploading a new patch which contains some classes which were missing from the earlier patch and also fixes the TODO pointed out by [~jpallas]. [~jpallas], yes, agreed. If there is an implementation which uses non-{{File}} based local replicas, the name of {{LocalReplicaInfo}} can be changed to something like {{FileReplicaInfo}}. However, that will be a class rename, and need not involve any more extensive changes. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti > Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382860#comment-15382860 ] Joe Pallas commented on HDFS-10636: --- I haven't reviewed every line, but I noticed in {{FsDatasetUtil.createNullChecksumFile}} there's a {{// TODO Auto-generated catch block}}. Overall I like this, and that it addresses some of the issues raised in HDFS-5194. One question about naming: In the context of HDFS-9806, the distinction between Provided and Local makes sense. But there might be other replica implementations that use non-{{File}} based storage but are still "local". I don't have a concrete alternative to "Local" to propose, though. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti > Attachments: HDFS-10636.001.patch > > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380342#comment-15380342 ] Virajith Jalaparti commented on HDFS-10636: --- The patch contains the following changes (copied here from comments in HDFS-9809). # Moving the {{java.io.File}} related APIs in {{ReplicaInfo}} ({{getBlockFile}}, {{getMetaFile}}) to a subclass of {{ReplicaInfo}} called {{LocalReplica}}. The classes {{FinalizedReplica}}, {{ReplicaInPipeline}}, {{ReplicaUnderRecovery}}, and {{ReplicaWaitingToBeRecovered}} are changed to be subclasses of {{LocalReplica}} instead of {{ReplicaInfo}}. The motivation behind this change is that we can have {{ReplicaInfo}} s that point to blocks located in remote stores and as a result don’t have associated {{java.io.File}} s. We added various functions to {{ReplicaInfo}} in order to replace the calls to {{ReplicaInfo#getBlockFile}}, and {{ReplicaInfo#getMetaFile}} in the rest of the code. # Using {{ReplicaInfo.getState()}} to get the state of a {{ReplicaInfo}} instead of using {{instanceof}}. A related change is to use the class {{ReplicaInfo}} to refer to the replica objects instead of the particular subclass (this required adding additional abstract functions to the {{ReplicaInfo}} class). # Addition of a {{ReplicaBuilder}} and replacing calls to the constructors of different {{ReplicaInfo}} subclasses ({{ReplicaInPipeline}}, {{ReplicaBeingWritten}}, etc.) with calls to the {{ReplicaBuilder}} with the appropriate parameters ({{ReplicaState}}, {{blockId}} etc.) set. # Changes related to {{ReplicaInPipeline}} * Change the {{ReplicaInPipeline}} to {{LocalReplicaInPipeline}}, and change {{ReplicaInPipelineInterface}} to {{ReplicaInPipeline}}. * Add a {{getReplicaInfo}} function to the (new) {{ReplicaInPipeline}} interface. * Move the functions related to writer threads ({{stopWriter}}, {{attemptToSetWriter}}, {{interruptThread}} and {{setWriter}}) to the new {{ReplicaInPipeline}} interface (i.e., the old {{ReplicaInPipelineInterface}}), as only {{ReplicaInPipeline}} objects will be associated with writer threads. The idea behind the changes above is to add a new {{ProvidedReplica}} class (an implementation of {{ReplicaInfo}}) which can be: (a) used to represent replicas stored in a provided storage (described in more detail in the design documentation of HDFS-9806). (b) treated as any other {{ReplicaInfo}} in the rest of the code. This would avoid changes to the rest of the Datanode as part of HDFS-9806. (c) written to using the existing replication pipeline, without implementing a separate write pipeline for HDFS-9806. > Modify ReplicaInfo to remove the assumption that replica metadata and data > are stored in java.io.File. > -- > > Key: HDFS-10636 > URL: https://issues.apache.org/jira/browse/HDFS-10636 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, fs >Reporter: Virajith Jalaparti > > Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the > definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be > present on external storages (HDFS-9806). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org