jojochuang commented on a change in pull request #2919: URL: https://github.com/apache/ozone/pull/2919#discussion_r772844808
########## File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/BaseFileChecksumHelper.java ########## @@ -0,0 +1,171 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.ozone.client; + +import com.google.common.annotations.VisibleForTesting; +import org.apache.hadoop.fs.FileChecksum; +import org.apache.hadoop.fs.MD5MD5CRC32GzipFileChecksum; +import org.apache.hadoop.hdds.scm.XceiverClientFactory; +import org.apache.hadoop.hdds.scm.XceiverClientSpi; +import org.apache.hadoop.io.DataOutputBuffer; +import org.apache.hadoop.io.MD5Hash; +import org.apache.hadoop.ozone.client.rpc.RpcClient; +import org.apache.hadoop.ozone.om.helpers.OmKeyArgs; +import org.apache.hadoop.ozone.om.helpers.OmKeyInfo; +import org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo; +import org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.util.List; + +/** + * The base class to support file checksum. + */ +public abstract class BaseFileChecksumHelper { + static final Logger LOG = + LoggerFactory.getLogger(BaseFileChecksumHelper.class); + + private OzoneVolume volume; + private OzoneBucket bucket; + private String keyName; + + private final long length; + protected RpcClient rpcClient; + protected XceiverClientFactory xceiverClientFactory; + + //private final Options.ChecksumCombineMode combineMode; + //private final BlockChecksumType blockChecksumType; Review comment: Yes. So basically Ozone EC design is pretty similar to HDFS EC. However, we don't have a separate request type to ask for the block checksum of an EC block group. A HDFS DN responds to a client blockGroupChecksum() request by issuing requests to other Datanodes to ask for block checksum of the internal blocks, and then the DataNode computes the checksum of the block group. Without a separate EC block group checksum lookup request, an Ozone client would need to request the block checksum of the internal blocks directly and compute the checksum of the block group at the client side. I am not that deep into the Ozone EC plan so this is what i plan given what i understand so far. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
