[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764197#comment-16764197
]
Surendra Singh Lilhore commented on HDFS-10636:
---
Thanks [~virajith], created new HDFS-14263 to fix this.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch,
> HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch,
> HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch,
> HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762972#comment-16762972
]
Virajith Jalaparti commented on HDFS-10636:
---
[~surendrasingh] agree. I think I added this as a check but it doesn't need to
be there.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch,
> HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch,
> HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch,
> HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762681#comment-16762681
]
Surendra Singh Lilhore commented on HDFS-10636:
---
[~virajith], I have one doubt for this patch.
Why do we need block file exists check for {{LocalReplica}} while opening input
stream for block ?
{code:java}
+ if(info != null && info.blockDataExists()) {
+ return info.getDataInputStream(seekOffset);
} else {
- try {
- return openAndSeek(blockFile, seekOffset);
- } catch (FileNotFoundException fnfe) {
{code}
I got {{requestShortCircuitFds()}} response little slower than 2.7.2. I checked
the code and found one extra {{File.exists()}} check added here.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch,
> HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch,
> HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch,
> HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16638560#comment-16638560
]
Íñigo Goiri commented on HDFS-10636:
I don't think Yetus for branch-2.9 works.
This is the patch we recently backported internally so +1.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
>Priority: Major
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636-branch-2.9.001.patch, HDFS-10636.001.patch,
> HDFS-10636.002.patch, HDFS-10636.003.patch, HDFS-10636.004.patch,
> HDFS-10636.005.patch, HDFS-10636.006.patch, HDFS-10636.007.patch,
> HDFS-10636.008.patch, HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15827020#comment-15827020
]
Andrew Wang commented on HDFS-10636:
Thanks [~virajith]! [~eddyxu] could you add a release note to this JIRA
explaining the potential incompatibility?
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826954#comment-15826954
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~andrew.wang], Yes, you are right in that all changes are to the Private
APIs in the datanode. However, [~eddyxu] mentioned that some vendors might have
implemented solutions based on {{FinalizedReplica}} (see
[this|https://issues.apache.org/jira/browse/HDFS-10636?focusedCommentId=15429061=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15429061]
comment). I think this was the reason the JIRA was marked incompatible.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15819714#comment-15819714
]
Andrew Wang commented on HDFS-10636:
Question, is this JIRA actually incompatible? I did a quick skim, and it only
seems to change things in the DN that are unannotated (thus defacto Private) or
annotated Private.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488318#comment-15488318
]
Virajith Jalaparti commented on HDFS-10636:
---
Great! Thanks [~eddyxu].
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[ https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488302#comment-15488302 ] Hudson commented on HDFS-10636: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10438 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10438/]) HDFS-10636. Modify ReplicaInfo to remove the assumption that replica (lei: rev 86c9862bec0248d671e657aa56094a2919b8ac14) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/RamDiskAsyncLazyPersistService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNodeFaultInjector.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/TestExternalDataset.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBeingWritten.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestCrcCorruption.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaUnderRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipelineInterface.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeRollingUpgrade.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTransferRbw.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInfo.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FinalizedReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestWriteToReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockPoolSliceStorage.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java * (edit)
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15487931#comment-15487931
]
Lei (Eddy) Xu commented on HDFS-10636:
--
Hi, [~virajith] Thanks for the insights.
bq. Our solution is to have a ProvidedReplica class which inherits ReplicaInfo,
and then have ProvidedFinalizedReplica inherit ProvidedReplica.
It makes sense now.
+1. Will commit shortly
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15486024#comment-15486024
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~eddyxu],
That is a very valid concern. Our solution is to have a {{ProvidedReplica}}
class which inherits {{ReplicaInfo}}, and then have
{{ProvidedFinalizedReplica}} inherit {{ProvidedReplica}}. Not only this, we
will need to have a hierarchy for {{ProvidedReplica}} that mirrors the
hierarchy for {{LocalReplica}} in the patch. This is somewhat forced on us as
different classes are used to implicitly implement the replica state machine
(for example, {{ReplicaInfo::getBytesOnDisk()}} must resolve correctly in
{{FsDatasetImpl}}). Thus, we will also need to have
{{ProvidedReplicaInPipeline}}, {{ProvidedReplicaUnderRecovery}} and others
which inherit {{ProvidedReplica}}.
One way to avoid implementing such shadow classes for {{ProvidedReplica}} would
be to explicitly implement a state machine for the replicas. This would be a
much more disruptive change, with significant amount of new code and changes
required to several parts of {{FsDatasetImpl}}, and {{FsVolumeImpl}}. The
current patch keeps most of the code paths intact with relatively minor changes
to the APIs of {{ReplicaInfo}}.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485831#comment-15485831
]
Lei (Eddy) Xu commented on HDFS-10636:
--
Sorry for late reply, [~chris] and [~virajith]
I think that my previous thoughts on types might put too much constraints and
make things difficult. Apologize for that.
This patch itself looks OK to me. +0. for the reason described below:
{code}
abstract public class LocalReplica extends ReplicaInfo {
}
// and
public class FinalizedReplica extends LocalReplica {
}
{code}
My concern is that in the future, where in the class hierarchy is
{{ProvidedFinalizedReplica}}, should it inherent from {{FinalizedReplica}} or
{{LocalReplica}}? In such cases, is {{ProvidedFinalizedReplica}} a
{{LocalReplica}}?
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485773#comment-15485773
]
Chris Douglas commented on HDFS-10636:
--
The latest version of the patch lgtm, +1. Thanks [~virajith] for seeing this
through.
[~eddyxu] do you have any other feedback on v10 before commit?
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478899#comment-15478899
]
Virajith Jalaparti commented on HDFS-10636:
---
The failed test case seems like a spurious failure (it runs fine on my dev
machine). The patch does not affect that code path.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch, HDFS-10636.008.patch,
> HDFS-10636.009.patch, HDFS-10636.010.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15478666#comment-15478666
]
Hadoop QA commented on HDFS-10636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 23s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 98m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10636 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827840/HDFS-10636.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c5c352b23e5c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cba973f |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16694/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16694/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16694/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16694/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
>
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475575#comment-15475575
]
Hadoop QA commented on HDFS-10636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 34s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10636 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827685/HDFS-10636.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7fa7e1d0fd5b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 011f3b2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/16684/artifact/patchprocess/patch-findbugs-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15472303#comment-15472303
]
Hadoop QA commented on HDFS-10636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 23 new + 856 unchanged - 94 fixed = 879 total (was 950) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 39s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10636 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827463/HDFS-10636.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2b335f745e06 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / d355573 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16663/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16663/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16663/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16663/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469060#comment-15469060
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~eddyxu],
bq. Should {{LocalReplica}} and {{ProvidedReplica}} extend {{FinalizedReplica}}?
Could you take a look at my previous [comment |
https://issues.apache.org/jira/browse/HDFS-10636?focusedCommentId=15433727=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15433727]
about this? I had a few questions as to what would be the motivation for this.
bq. {{ReplicaBuilder#buildLocalReplicaInPipeline}} can be private.
This will not be sufficient as the function is used in {{FsVolumeImpl}}.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468790#comment-15468790
]
Lei (Eddy) Xu commented on HDFS-10636:
--
Hi, [~virajith], Thanks for the updates. And thanks for [~jpallas]'s inputs.
I have a few questions remained:
* Should {{LocalReplica}} and {{ProvidedReplica}} extend {{FinalizedReplica}}?
It seems the opposite way in the current patch. Similarly concerns for
{{ReplicaBeingWritten}} and {{LocalReplicaBeingWritten}}.
* {{ReplicaBuilder#buildLocalReplicaInPipeline}} can be private.
The rest looks good to me.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch,
> HDFS-10636.006.patch, HDFS-10636.007.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468517#comment-15468517
]
Hadoop QA commented on HDFS-10636:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 25 new + 856 unchanged - 94 fixed = 881 total (was 950) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 94m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDNFencing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10636 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826989/HDFS-10636.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3e2ad5721046 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f6c0b75 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/whitespace-tabs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16650/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433904#comment-15433904
]
Joe Pallas commented on HDFS-10636:
---
Possibly the only requirement to subclass {{FinalizedReplica}} today is that it
appears in the dataset SPI, so any alternative implementation has to use it.
The proposal removes that, so maybe there isn't a strong reason to have it be a
type (if I understand the distinction I think [~virajith] is making between
class as type and class as state). There are several places in the current
patch where {{FinalizedReplica}} becomes {{ReplicaInfo}}, and I think that's a
reasonable change.
The concern [~eddyxu] expressed above about states and types is valid, but, in
this case, I think the type system isn't really helping the programmer very
much. Existing code using {{FinalizedReplica}} is already frequently stuck
with only {{ReplicaInfo}} known at compile time, so a state check is needed
anyway, and using a cast to {{FinalizedReplica}} after the state check doesn't
add much.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433727#comment-15433727
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~eddyxu],
bq. I know vendors who have already implemented similar solutions based on
{{FinalizedReplica}}.
I am trying to understand the use case for {{FinalizedReplica}} you mentioned.
What properties of {{FinalizedReplica}} do you think we should preserve for
these implementations to continue working? This question is also related to the
following.
bq. Have you considered about making {{LocalReplica}} and {{ProvidedReplica}}
being subclasses of {{FinalizedReplica}}?
{{FinalizedReplica}}, as implemented today, does not have any particular
characteristics other than it's state and type. There do not exist functions in
{{FinalizedReplica}} that are not in {{ReplicaInfo}}. Is the goal of your
proposal to have {{FinalizedReplica}} used as a type, and not just state? Do
you see having to do this for other states as well (RUR, RBW, RWR, TEMPORARY)?
or having it for {{FinalizedReplica}} is more important?
One concrete proposal for this can be the following: {{FinalizedReplica}} will
be an abstract subclass of {{ReplicaInfo}} (overrides
{{ReplicaInfo::getState()}}). We can then define {{LocalFinalizedReplica}} and
{{ProvidedFinalizedReplica}}, which can be subclasses of {{FinalizedReplica}}.
However, this would cause {{LocalFinalizedReplica}} to re-implement the
functionality of other {{LocalReplica}}s.
One way to avoid such re-implementation could be to have {{FinalizedReplica}}
as an interface, with a {{FinalizedReplica::getReplicaInfo()}} function to
perform all the {{ReplicaInfo}} related functions. Then
{{LocalFinalizedReplica}} can subclass {{LocalReplica}} and implement the
{{FinalizedReplica}} interface.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15429061#comment-15429061
]
Lei (Eddy) Xu commented on HDFS-10636:
--
bq. If not, are there existing cases where vendors might be using
FinalizedReplica for non-File backed data?
Yea. I know vendors who have already implemented similar solutions based on
{{FinalizedReplica}}, because {{ProvidedReplica}} did not exist back to the
days. But I think it might be OK for them if this patch did not break the
their code.
Another option might be that, "Finalized" looks like a state in the pipeline
state machine, instead of a "Location" where the replica is stored, IMO. Have
you considered about making {{LocalReplica}} and {{ProvidedReplica}} being
subclasses of {{FinalizedReplica}}? So that for the pipeline state machine part
of code, all the existing code can re-use the same {{FinalizedReplica}} as what
it is today. Would love to hear your thoughts about it.
bq. Having a good test coverage seems a cleaner way to solve this.
Agree. That would be a nice thing to do.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427479#comment-15427479
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~eddyxu],
bq. E.g., If you implement recovery logic in Azure or S3, the input should not
be a File based replica.
Good point. I agree with this. Having {{StorageLocation}} instead of {{File}}
makes sense (other parts of the patch aim to already do this). I will make the
necessary changes.
bq. Would that still be the case for Azure?
For HDFS-9806, the idea would be to have a {{ProvidedReplica}} which will be
used to refer to data in external storages. As shown in HDFS-10675, this class
would implement {{ReplicaInfo}} and {{ReplicaInPipeline}}, so that we can
re-use the existing code for building replication pipelines and go through the
block life cycle.
bq. so I guess many vendors might expect using {{FinalizedReplica}} to directly
represent the block data.
Are you suggesting this in the context of HDFS-9806? If so, I think they should
be using {{ProvidedReplica}} (as mentioned in my previous point). If not, are
there existing cases where vendors might be using {{FinalizedReplica}} for
non-{{File}} backed data?
One way to deal with this issue with {{FinalizedReplica}} and the usage of
class types is to have abstract classes for each replica type, and then have
implementations for local and provided replicas. This will end up having a lot
of classes which are essentially {{ReplicaInfo}}. Having a good test coverage
seems a cleaner way to solve this. Thoughts?
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15427349#comment-15427349
]
Lei (Eddy) Xu commented on HDFS-10636:
--
Hi, [~virajith]
Thanks a lot for update the patch and address the comments. And appreciate
much for taking our comments into consideration.
bq. The other way to deal with this would be to add setDir to ReplicaInfo but
to avoid File as a parameter.
What I am worried about is that, this constraint might not hold true for wider
audience. E.g., If you implement recovery logic in Azure or S3, the input
should not be a File based replica.
bq.it should take in an URI. Which do you think is better?
Maybe using {{StorageLocation}} as input? There was some efforts to abstract
{{File}} to {{StorageLocation}}.
bq. {{FinalizedReplica}} in the current type hierarchy, is a LocalReplica.
Would that still be the case for Azure? I have saw a few cases that it is not a
file-based solution. Also, {{FinalizedReplica}} should be the most common class
in the {{ReplicaInfo}} hierarchy, so I guess many vendors might expect using
{{FinalizedReplica}} to directly represent the block data.
bq. The way I was thinking about it was that the state of ReplicaInfo should be
known using ReplicaInfo::getState(), and not using the type system.
I don't have strong option against the solution proposed here. The concerns of
getting rid of type system and using {int} / {enum} to indicate the class type
might make the other developer difficult to have good confident to write
correct code, because like dynamic language as python/ruby, the bugs become
hard to spot during compile time. If we can come up a good test coverage, the
concern can be overcome to some extend.
Thanks.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419293#comment-15419293
]
Virajith Jalaparti commented on HDFS-10636:
---
Hi [~eddyxu],
Thank you for the comments.
I agree with points 1 and 2, and will fix them.
bq. {{breakHardlinksIfNeeded}}, {{copyMetadata}} and {{copyBlockdata}} should
not be in {{ReplicaInfo}}. Or should not use {{File}} as input.
Agree that {{copyMetadata}} and {{copyBlockdata}} should not have {{File}} as a
parameter. I will change it to {{URI}} to be more general.
{{breakHardLinksIfNeeded}} has always been in {{ReplicaInfo}}. I made it
abstract and moved the implementation to {{LocalReplica}}.
bq. {{ReplicaUnderRecovery}}. Is there a way to avoid casting {{ReplicaInfo}}
to {{LocalReplica}}?
The only place where the fact that {{original}} is a {{LocalReplica}} matters
is in {{ReplicaUnderRecovery::setDir()}}. One way to address this would be to
add the cast only when {{original.setDir()}} is called. The other way to deal
with this would be to add {{setDir}} to {{ReplicaInfo}} but to avoid {{File}}
as a parameter, it should take in an URI. Which do you think is better?
bq. In general, there are places in this patch that return {{ReplicaInfo}} for
{{FinalizedReplica}}. which would makes type system weaker and is not
future-proof. Is it necessary to be changed?
This was intentional. The way I was thinking about it was that the state of
{{ReplicaInfo}} should be known using {{ReplicaInfo::getState()}}, and not
using the type system. The current code does the latter -- it uses the type
system to ensure that replicas are in a certain state. Not relying on the type
system and using the former (use {{ReplicaInfo::getState()}}) seems a cleaner
way of doing this. What do you think? Also, {{FinalizedReplica}} in the current
type hierarchy, is a {{LocalReplica}}. So, referring to replicas using
{{FinalizedReplica}} assumes that they are {{LocalReplica}} and hence, backed
by {{File}} s.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418219#comment-15418219
]
Lei (Eddy) Xu commented on HDFS-10636:
--
Hi, [~virajith]
Thanks for working on this. A few questions.
* In {{FsDatasetImpl#fetchReplication}} , please remove the usage {{LOG.info}}.
* In {{ReplicaInfo.java}}. The following comments seem irrelevant to the code
below.
{code}
/** directory where block & meta files belong. */
{code}
Also please do not remove the comments for the constroctors.
* {{breakHardlinksIfNeeded}}, {{ copyMetadata}} and {{copyBlockdata}} should
not be in {{ReplicaInfo}}. Or should not use {{File}} as input.
* {{ReplicaUnderRecovery}}. Is there a way to avoid casting {{ReplicaInfo}} to
{{LocalReplica}}?
* In general, there are places in this patch that return {{ReplicaInfo}} for
{{FinalizedReplica}}. which would makes type system weaker and is not
future-proof. Is it necessary to be changed?
Thanks.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch, HDFS-10636.005.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15408705#comment-15408705
]
Virajith Jalaparti commented on HDFS-10636:
---
Thanks [~chris.douglas] for the comments and fixing the checkstyle/findbugs.
I am attaching a new patch to address the above comments.
bq. Breaking up the switch statement in build() avoids a checkstyle complaint?
This might be more legible if each (fairly long) case stmt were in a private
method.
Addressed in the patch -- moved each case statement to a private function.
bq. Is {{DataStorage::getTrashDirectoryForBlockFile}} now unused, except by
tests?
Yes, it is no longer used/required. I modified the tests that access
{{DataStorage::getTrashDirectoryForBlockFile}}, to ensure that it is no longer
needed and removed this function. I also modified tests that access
{{BlockPoolSliceStorage::getTrashDirectory(File)}} to access
{{BlockPoolSliceStorage::getTrashDirectory(ReplicaInfo)}}, and made the former
a private function in {{BlockPoolSliceStorage}}.
bq. the call to {{Storage::nativeCopyFileUnbuffered}} is removed, so there's no
path to the {{NativeIO}} libraries. This should be maintained.
Agree. I have restored this by adding two new abstract functions to
{{ReplicaInfo}}: {{copyMetadata}} and {{copyBlockdata}}, and implementing them
in {{LocalReplica}} to use {{Storage::nativeCopyFileUnbuffered}}.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>Assignee: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch,
> HDFS-10636.003.patch, HDFS-10636.004.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15383319#comment-15383319
]
Virajith Jalaparti commented on HDFS-10636:
---
Uploading a new patch which contains some classes which were missing from the
earlier patch and also fixes the TODO pointed out by [~jpallas].
[~jpallas], yes, agreed. If there is an implementation which uses non-{{File}}
based local replicas, the name of {{LocalReplicaInfo}} can be changed to
something like {{FileReplicaInfo}}. However, that will be a class rename, and
need not involve any more extensive changes.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch, HDFS-10636.002.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15382860#comment-15382860
]
Joe Pallas commented on HDFS-10636:
---
I haven't reviewed every line, but I noticed in
{{FsDatasetUtil.createNullChecksumFile}} there's a {{// TODO Auto-generated
catch block}}.
Overall I like this, and that it addresses some of the issues raised in
HDFS-5194.
One question about naming: In the context of HDFS-9806, the distinction
between Provided and Local makes sense. But there might be other replica
implementations that use non-{{File}} based storage but are still "local". I
don't have a concrete alternative to "Local" to propose, though.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
> Attachments: HDFS-10636.001.patch
>
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10636) Modify ReplicaInfo to remove the assumption that replica metadata and data are stored in java.io.File.
[
https://issues.apache.org/jira/browse/HDFS-10636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380342#comment-15380342
]
Virajith Jalaparti commented on HDFS-10636:
---
The patch contains the following changes (copied here from comments in
HDFS-9809).
# Moving the {{java.io.File}} related APIs in {{ReplicaInfo}}
({{getBlockFile}}, {{getMetaFile}}) to a subclass of {{ReplicaInfo}} called
{{LocalReplica}}. The classes {{FinalizedReplica}}, {{ReplicaInPipeline}},
{{ReplicaUnderRecovery}}, and {{ReplicaWaitingToBeRecovered}} are changed to be
subclasses of {{LocalReplica}} instead of {{ReplicaInfo}}. The motivation
behind this change is that we can have {{ReplicaInfo}} s that point to blocks
located in remote stores and as a result don’t have associated {{java.io.File}}
s.
We added various functions to {{ReplicaInfo}} in order to replace the calls to
{{ReplicaInfo#getBlockFile}}, and {{ReplicaInfo#getMetaFile}} in the rest of
the code.
# Using {{ReplicaInfo.getState()}} to get the state of a {{ReplicaInfo}}
instead of using {{instanceof}}. A related change is to use the class
{{ReplicaInfo}} to refer to the replica objects instead of the particular
subclass (this required adding additional abstract functions to the
{{ReplicaInfo}} class).
# Addition of a {{ReplicaBuilder}} and replacing calls to the constructors of
different {{ReplicaInfo}} subclasses ({{ReplicaInPipeline}},
{{ReplicaBeingWritten}}, etc.) with calls to the {{ReplicaBuilder}} with the
appropriate parameters ({{ReplicaState}}, {{blockId}} etc.) set.
# Changes related to {{ReplicaInPipeline}}
* Change the {{ReplicaInPipeline}} to {{LocalReplicaInPipeline}}, and change
{{ReplicaInPipelineInterface}} to {{ReplicaInPipeline}}.
* Add a {{getReplicaInfo}} function to the (new) {{ReplicaInPipeline}}
interface.
* Move the functions related to writer threads ({{stopWriter}},
{{attemptToSetWriter}}, {{interruptThread}} and {{setWriter}}) to the new
{{ReplicaInPipeline}} interface (i.e., the old {{ReplicaInPipelineInterface}}),
as only {{ReplicaInPipeline}} objects will be associated with writer threads.
The idea behind the changes above is to add a new {{ProvidedReplica}} class (an
implementation of {{ReplicaInfo}}) which can be:
(a) used to represent replicas stored in a provided storage (described in more
detail in the design documentation of HDFS-9806).
(b) treated as any other {{ReplicaInfo}} in the rest of the code. This would
avoid changes to the rest of the Datanode as part of HDFS-9806.
(c) written to using the existing replication pipeline, without implementing a
separate write pipeline for HDFS-9806.
> Modify ReplicaInfo to remove the assumption that replica metadata and data
> are stored in java.io.File.
> --
>
> Key: HDFS-10636
> URL: https://issues.apache.org/jira/browse/HDFS-10636
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: datanode, fs
>Reporter: Virajith Jalaparti
>
> Replace java.io.File related APIs from {{ReplicaInfo}}, and enable the
> definition of new {{ReplicaInfo}} sub-classes whose metadata and data can be
> present on external storages (HDFS-9806).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
