[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455364#comment-16455364 ] Hudson commented on HDFS-12853: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-12853. Ozone: Optimize chunk writes for Ratis by avoiding double (omalley: rev 3965f1ec9909833bab2d16be6561fa63ed56038f) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/Dispatcher.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ChunkManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/metrics/TestContainerMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/ScmConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/DatanodeContainerProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/ChunkManagerImpl.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch, > HDFS-12853-HDFS-7240.002.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451148#comment-16451148 ] Hudson commented on HDFS-12853: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-12853. Ozone: Optimize chunk writes for Ratis by avoiding double (szetszwo: rev 886b2cb6e797ab7d742dc2ceaeff818ee656b6cc) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/ScmConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/common/impl/TestContainerPersistence.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/ozone/OzoneConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/DatanodeContainerProtocol.proto * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/Dispatcher.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/interfaces/ChunkManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/ozone/container/metrics/TestContainerMetrics.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/XceiverServerRatis.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/impl/ChunkManagerImpl.java > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh >Priority: Major > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch, > HDFS-12853-HDFS-7240.002.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316034#comment-16316034 ] Tsz Wo Nicholas Sze commented on HDFS-12853: Oops, I found it now. > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch, > HDFS-12853-HDFS-7240.002.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316033#comment-16316033 ] Tsz Wo Nicholas Sze commented on HDFS-12853: writeChunkMap not found in the 002 patch. Wrong file? > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch, > HDFS-12853-HDFS-7240.002.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[
https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316022#comment-16316022
]
Mukul Kumar Singh commented on HDFS-12853:
--
If we don't have the synchronization mentioned the above, it seems that the
patch won't work?
bq. Yes, thats correct, without the synchronization, the fix will not work.
However the following lines in the latest patch adds the syncronization
{code}
if (requestProto.getCmdType() == ContainerProtos.Type.WriteChunk) {
WriteChunkRequestProto write = requestProto.getWriteChunk();
CompletableFuture stateMachineFuture =
writeChunkMap.remove(write.getChunkData().getChunkName());
return stateMachineFuture
.thenComposeAsync(v ->
CompletableFuture.completedFuture(runCommand(requestProto)));
{code}
> Ozone: Optimize chunk writes for Ratis by avoiding double writes
>
>
> Key: HDFS-12853
> URL: https://issues.apache.org/jira/browse/HDFS-12853
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12853-HDFS-7240.001.patch,
> HDFS-12853-HDFS-7240.002.patch
>
>
> Ozone, in replicated mode writes the data twice, once to the raft log and
> then to the statemachine.
> This results in the data being written twice during a particular chunk write,
> this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized
> by writing the data to the statemachine only once.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316006#comment-16316006 ] Tsz Wo Nicholas Sze commented on HDFS-12853: > COMMIT_DATA should be sync'ed with WRITE_DATA so that COMMIT_DATA is always > after WRITE_DATA. If we don't have the synchronization mentioned the above, it seems that the patch won't work? > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch, > HDFS-12853-HDFS-7240.002.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[
https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314910#comment-16314910
]
genericqa commented on HDFS-12853:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
16s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
27s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
34s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
21s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project: The patch generated 4 new +
0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
9s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}136m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}218m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeysRatis |
| | hadoop.ozone.ozShell.TestOzoneShell |
| | hadoop.ozone.TestOzoneConfigurationFields |
| | hadoop.ozone.container.common.impl.TestContainerPersistence |
| | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.ksm.TestKeySpaceManager |
| | hadoop.ozone.scm.TestSCMCli |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d11161b |
| JIRA Issue | HDFS-12853 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[ https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312623#comment-16312623 ] Tsz Wo Nicholas Sze commented on HDFS-12853: - Also, we need to delete the tmp files in case of failure. > Ozone: Optimize chunk writes for Ratis by avoiding double writes > > > Key: HDFS-12853 > URL: https://issues.apache.org/jira/browse/HDFS-12853 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Mukul Kumar Singh >Assignee: Mukul Kumar Singh > Fix For: HDFS-7240 > > Attachments: HDFS-12853-HDFS-7240.001.patch > > > Ozone, in replicated mode writes the data twice, once to the raft log and > then to the statemachine. > This results in the data being written twice during a particular chunk write, > this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized > by writing the data to the statemachine only once. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12853) Ozone: Optimize chunk writes for Ratis by avoiding double writes
[
https://issues.apache.org/jira/browse/HDFS-12853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312614#comment-16312614
]
Tsz Wo Nicholas Sze commented on HDFS-12853:
Thanks for working on this. The patch looks good in general. Some comments:
- COMMIT_DATA should be sync'ed with WRITE_DATA so that COMMIT_DATA is always
after WRITE_DATA.
- OZONE_SCM_RATIS_NUM_WRITE_CHUNK_THREADS is a container conf so that it should
be moved to OzoneConfigKeys.DFS_CONTAINER_RATIS_...
- Let's change the format from "TMP--" to
"..tmp" so that it is easier for admin to do ls.
- Pass the original proto as a prototype to newBuilder. Then, we only need to
change the new fields.
{code}
public TransactionContext startTransaction(RaftClientRequest request)
throws IOException {
final org.apache.ratis.shaded.com.google.protobuf.ByteString content =
request.getMessage().getContent();
final ContainerCommandRequestProto proto =
ContainerCommandRequestProto.parseFrom(ShadedProtoUtil.asByteString(content));
final SMLogEntryProto log;
if (proto.getCmdType() == ContainerProtos.Type.WriteChunk) {
final ContainerProtos.WriteChunkRequestProto write =
proto.getWriteChunk();
final ContainerProtos.WriteChunkRequestProto dataWriteProto =
ContainerProtos.WriteChunkRequestProto.newBuilder(write)
.setStage(ContainerProtos.Stage.WRITE_DATA)
.build();
ContainerCommandRequestProto dataProto = ContainerCommandRequestProto
.newBuilder(proto)
.setWriteChunk(dataWriteProto)
.build();
org.apache.ratis.shaded.com.google.protobuf.ByteString dataByteString =
ShadedProtoUtil.asShadedByteString(dataProto.toByteArray());
final ContainerProtos.WriteChunkRequestProto commitWriteProto =
ContainerProtos.WriteChunkRequestProto.newBuilder(write)
.setStage(ContainerProtos.Stage.COMMIT_DATA)
.build();
ContainerCommandRequestProto commitProto = ContainerCommandRequestProto
.newBuilder(proto)
.setWriteChunk(commitWriteProto)
.build();
org.apache.ratis.shaded.com.google.protobuf.ByteString commitByteString =
ShadedProtoUtil.asShadedByteString(commitProto.toByteArray());
log = SMLogEntryProto.newBuilder()
.setData(commitByteString)
.setStateMachineData(dataByteString)
.build();
} else {
log = SMLogEntryProto.newBuilder().setData(content).build();
}
return new TransactionContext(this, request, log);
}
{code}
Other comments (all can be done separately):
- ChunkUtils.writeData should use CREATE_NEW instead of CREATE.
- ChunkUtils.writeData, verifyChecksum should not throw
NoSuchAlgorithmException. Just wrap it as IllegalArgumentException. Also use
and ThreadLocal for it.
{code}
//ChunkUtils
private static final ThreadLocal FILE_MESSAGE_DIGEST
= ThreadLocal.withInitial(() -> {
try {
return MessageDigest.getInstance(OzoneConsts.FILE_HASH);
} catch (NoSuchAlgorithmException e) {
throw new IllegalArgumentException("Failed to create MessageDigest
for "
+ OzoneConsts.FILE_HASH, e);
}
});
public static MessageDigest getMessageDigest() {
final MessageDigest md = FILE_MESSAGE_DIGEST.get();
md.reset();
return md;
}
{code}
- Pass data as ByteString instead of byte[] in order to avoid data copying.
- File.length() calls require local fs access so that length for the same file
should be stored.
> Ozone: Optimize chunk writes for Ratis by avoiding double writes
>
>
> Key: HDFS-12853
> URL: https://issues.apache.org/jira/browse/HDFS-12853
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
> Fix For: HDFS-7240
>
> Attachments: HDFS-12853-HDFS-7240.001.patch
>
>
> Ozone, in replicated mode writes the data twice, once to the raft log and
> then to the statemachine.
> This results in the data being written twice during a particular chunk write,
> this is subobtimal. With RATIS-122 the statemachine in Ozone can be optimized
> by writing the data to the statemachine only once.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
