szetszwo commented on code in PR #8541:
URL: https://github.com/apache/ozone/pull/8541#discussion_r2153248165
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDirectoryInfo.java:
##########
@@ -208,20 +208,14 @@ public DirectoryInfo getProtobuf() {
return pib.build();
}
- /**
- * Parses DirectoryInfo protobuf and creates OmPrefixInfo.
- * @param dirInfo
- * @return instance of OmDirectoryInfo
- */
- public static OmDirectoryInfo getFromProtobuf(DirectoryInfo dirInfo) {
+ public static Builder newBuilderFromProtobufPartial(DirectoryInfo dirInfo) {
Review Comment:
Change it similarly as `OmBucketInfo`, i.e. use one method to
include/exclude large fields.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java:
##########
@@ -595,18 +595,10 @@ public static OmBucketInfo getFromProtobuf(BucketInfo
bucketInfo) {
return getFromProtobuf(bucketInfo, null);
}
- /**
- * Parses BucketInfo protobuf and creates OmBucketInfo.
- * @param bucketInfo
- * @return instance of OmBucketInfo
- */
- public static OmBucketInfo getFromProtobuf(BucketInfo bucketInfo,
- BucketLayout buckLayout) {
- Builder obib = OmBucketInfo.newBuilder()
+ public static Builder newBuilderFromProtobufPartial(BucketInfo bucketInfo) {
Review Comment:
- Handle all fields in one method follow the same order as in the proto.
- Add a boolean to include/exclude the large fields.
- Add javadoc.
```java
/**
* Create a builder from the given proto.
* @param includeLargeFields Omitted the large fields: acl, metadata,
beinfo.
*/
public static Builder newBuilder(BucketInfo bucketInfo, boolean
includeLargeFields) {
final Builder builder = OmBucketInfo.newBuilder()
.setVolumeName(bucketInfo.getVolumeName())
.setBucketName(bucketInfo.getBucketName());
if (includeLargeFields) {
// acl
builder.setAcls(bucketInfo.getAclsList().stream().map(
OzoneAcl::fromProtobuf).collect(Collectors.toList()));
}
builder.setIsVersionEnabled(bucketInfo.getIsVersionEnabled())
.setStorageType(StorageType.valueOf(bucketInfo.getStorageType()))
.setCreationTime(bucketInfo.getCreationTime());
if (includeLargeFields) {
// metadata
builder.addAllMetadata(KeyValueUtil.getFromProtobuf(bucketInfo.getMetadataList()));
if (bucketInfo.hasBeinfo()) {
// beinfo
builder.setBucketEncryptionKey(OMPBHelper.convert(bucketInfo.getBeinfo()));
}
}
if (bucketInfo.hasObjectID()) {
builder.setObjectID(bucketInfo.getObjectID());
}
if (bucketInfo.hasUpdateID()) {
builder.setUpdateID(bucketInfo.getUpdateID());
}
builder.setModificationTime(bucketInfo.getModificationTime());
if (bucketInfo.hasSourceVolume()) {
builder.setSourceVolume(bucketInfo.getSourceVolume());
}
if (bucketInfo.hasSourceBucket()) {
builder.setSourceBucket(bucketInfo.getSourceBucket());
}
builder.setUsedBytes(bucketInfo.getUsedBytes())
.setQuotaInBytes(bucketInfo.getQuotaInBytes())
.setQuotaInNamespace(bucketInfo.getQuotaInNamespace())
.setUsedNamespace(bucketInfo.getUsedNamespace())
if (bucketInfo.hasBucketLayout()) {
builder.setBucketLayout(BucketLayout.fromProto(bucketInfo.getBucketLayout()));
}
if (bucketInfo.hasOwner()) {
builder.setOwner(bucketInfo.getOwner());
}
if (bucketInfo.hasDefaultReplicationConfig()) {
builder.setDefaultReplicationConfig(
DefaultReplicationConfig.fromProto(bucketInfo.getDefaultReplicationConfig()));
}
return builder;
}
```
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java:
##########
@@ -730,18 +731,18 @@ private KeyInfo getProtobuf(boolean ignorePipeline,
String fullKeyName,
return kb.build();
}
- public static OmKeyInfo getFromProtobuf(KeyInfo keyInfo) {
+ public static Builder newBuilderFromProtobufPartial(KeyInfo keyInfo) {
if (keyInfo == null) {
return null;
}
List<OmKeyLocationInfoGroup> omKeyLocationInfos = new ArrayList<>();
- for (KeyLocationList keyLocationList : keyInfo.getKeyLocationListList()) {
+ for (OzoneManagerProtocolProtos.KeyLocationList keyLocationList :
keyInfo.getKeyLocationListList()) {
Review Comment:
Revert this change.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDirectoryInfo.java:
##########
@@ -208,20 +208,14 @@ public DirectoryInfo getProtobuf() {
return pib.build();
}
- /**
- * Parses DirectoryInfo protobuf and creates OmPrefixInfo.
- * @param dirInfo
- * @return instance of OmDirectoryInfo
- */
- public static OmDirectoryInfo getFromProtobuf(DirectoryInfo dirInfo) {
+ public static Builder newBuilderFromProtobufPartial(DirectoryInfo dirInfo) {
OmDirectoryInfo.Builder opib = OmDirectoryInfo.newBuilder()
- .setName(dirInfo.getName())
- .setCreationTime(dirInfo.getCreationTime())
- .setModificationTime(dirInfo.getModificationTime())
- .setAcls(OzoneAclUtil.fromProtobuf(dirInfo.getAclsList()));
+ .setName(dirInfo.getName())
+ .setCreationTime(dirInfo.getCreationTime())
+ .setModificationTime(dirInfo.getModificationTime());
if (dirInfo.getMetadataList() != null) {
Review Comment:
Do we want to include metadata in this case?
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ReconOMDBDefinition.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.codec;
+
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.BUCKET_TABLE;
+import static
org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_DIR_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_KEY_TABLE;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.hdds.utils.db.StringCodec;
+import org.apache.hadoop.ozone.ClientVersion;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.codec.OMDBDefinition;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Recon-specific OM RocksDB schema definition.
+ *
+ * <p>
+ * This is a specialized version of {@code
org.apache.hadoop.ozone.om.codec.OMDBDefinition},
+ * used by the Recon service (the Ozone analytics layer). It uses lightweight,
+ * “partial” Protobuf codecs for faster deserialization and to skip loading
+ * ACL or permission details unnecessary for analytic workloads.
+ * </p>
+ *
+ * <p><b>Included Column Families:</b></p>
+ * <ul>
+ * <li>BUCKET_TABLE_DEF: maps volume/bucket → {@link OmBucketInfo}</li>
+ * <li>KEY_TABLE_DEF, OPEN_KEY_TABLE_DEF: map volume/bucket/key* → {@link
OmKeyInfo}</li>
+ * <li>DELETED_TABLE_DEF: maps deleted keys → {@link RepeatedOmKeyInfo}</li>
+ * <li>FILE_TABLE_DEF, OPEN_FILE_TABLE_DEF, DIRECTORY_TABLE_DEF,
DELETED_DIR_TABLE_DEF:
+ * FSO layout support → {@link OmDirectoryInfo}, {@link OmKeyInfo}</li>
+ * <li>… and all other column families imported from the canonical OM
definition, such as:
+ * DELEGATION_TOKEN_TABLE_DEF, META_TABLE_DEF, S3/tenant/snapshot
tables, etc.</li>
+ * </ul>
+ *
+ * <p>
+ * The column families are initialized via {@link DBColumnFamilyDefinition}
and passed
+ * into the parent {@link DBDefinition.WithMap} to initialize Recon's RocksDB
instance.
+ * </p>
+ *
+ * <p><b>Codecs:</b></p>
+ * <ul>
+ * <li>Custom {@link DelegatedCodec} are used to transform partial protobuf
messages into OM helper objects.</li>
+ * <li>Examples include {@link OmBucketInfo#getCodec}-based code reuse for
Om-only read path.</li>
+ * </ul>
+ *
+ * <p><b>Singleton Access:</b></p>
+ * This class implements the singleton pattern:
+ * <pre>
+ * ReconOMDBDefinition def = ReconOMDBDefinition.get();
+ * </pre>
+ *
+ */
+public final class ReconOMDBDefinition extends DBDefinition.WithMap {
+
+ //---------------------------------------------------------------------------
+ // Volume, Bucket, Prefix and Transaction Tables:
Review Comment:
The javadoc is invalid. Please update it.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDirectoryInfo.java:
##########
@@ -235,6 +229,17 @@ public static OmDirectoryInfo
getFromProtobuf(DirectoryInfo dirInfo) {
if (dirInfo.hasOwnerName()) {
opib.setOwner(dirInfo.getOwnerName());
}
+ return opib;
+ }
+
+ /**
+ * Parses DirectoryInfo protobuf and creates OmPrefixInfo.
+ * @param dirInfo
+ * @return instance of OmDirectoryInfo
+ */
Review Comment:
This javadoce does not provide any useful information and generate "tag
description is missing" warning. Please remove it or rewrite it.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmDirectoryInfo.java:
##########
@@ -208,20 +208,14 @@ public DirectoryInfo getProtobuf() {
return pib.build();
}
- /**
- * Parses DirectoryInfo protobuf and creates OmPrefixInfo.
- * @param dirInfo
- * @return instance of OmDirectoryInfo
- */
- public static OmDirectoryInfo getFromProtobuf(DirectoryInfo dirInfo) {
+ public static Builder newBuilderFromProtobufPartial(DirectoryInfo dirInfo) {
OmDirectoryInfo.Builder opib = OmDirectoryInfo.newBuilder()
- .setName(dirInfo.getName())
- .setCreationTime(dirInfo.getCreationTime())
- .setModificationTime(dirInfo.getModificationTime())
- .setAcls(OzoneAclUtil.fromProtobuf(dirInfo.getAclsList()));
+ .setName(dirInfo.getName())
+ .setCreationTime(dirInfo.getCreationTime())
+ .setModificationTime(dirInfo.getModificationTime());
if (dirInfo.getMetadataList() != null) {
opib.addAllMetadata(KeyValueUtil
- .getFromProtobuf(dirInfo.getMetadataList()));
+ .getFromProtobuf(dirInfo.getMetadataList()));
Review Comment:
Combine two lines.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmBucketInfo.java:
##########
@@ -615,22 +607,48 @@ public static OmBucketInfo getFromProtobuf(BucketInfo
bucketInfo,
.setQuotaInBytes(bucketInfo.getQuotaInBytes())
.setUsedNamespace(bucketInfo.getUsedNamespace())
.setQuotaInNamespace(bucketInfo.getQuotaInNamespace());
- if (buckLayout != null) {
- obib.setBucketLayout(buckLayout);
- } else if (bucketInfo.getBucketLayout() != null) {
- obib.setBucketLayout(
- BucketLayout.fromProto(bucketInfo.getBucketLayout()));
- }
+
if (bucketInfo.hasDefaultReplicationConfig()) {
- obib.setDefaultReplicationConfig(
- DefaultReplicationConfig.fromProto(
- bucketInfo.getDefaultReplicationConfig()));
+ builder.setDefaultReplicationConfig(
+
DefaultReplicationConfig.fromProto(bucketInfo.getDefaultReplicationConfig()));
}
if (bucketInfo.hasObjectID()) {
- obib.setObjectID(bucketInfo.getObjectID());
+ builder.setObjectID(bucketInfo.getObjectID());
}
if (bucketInfo.hasUpdateID()) {
- obib.setUpdateID(bucketInfo.getUpdateID());
+ builder.setUpdateID(bucketInfo.getUpdateID());
+ }
+ if (bucketInfo.hasSourceVolume()) {
+ builder.setSourceVolume(bucketInfo.getSourceVolume());
+ }
+ if (bucketInfo.hasSourceBucket()) {
+ builder.setSourceBucket(bucketInfo.getSourceBucket());
+ }
+ if (bucketInfo.hasOwner()) {
+ builder.setOwner(bucketInfo.getOwner());
+ }
+ if (bucketInfo.hasBucketLayout()) {
+
builder.setBucketLayout(BucketLayout.fromProto(bucketInfo.getBucketLayout()));
+ }
+
+ return builder;
+ }
+
+ /**
+ * Parses BucketInfo protobuf and creates OmBucketInfo.
+ * @param bucketInfo
+ * @return instance of OmBucketInfo
+ */
Review Comment:
This javadoce does not provide any useful information and generate "tag
description is missing" warning. Please remove it or rewrite it.
##########
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/helpers/OmKeyInfo.java:
##########
@@ -730,18 +731,18 @@ private KeyInfo getProtobuf(boolean ignorePipeline,
String fullKeyName,
return kb.build();
}
- public static OmKeyInfo getFromProtobuf(KeyInfo keyInfo) {
+ public static Builder newBuilderFromProtobufPartial(KeyInfo keyInfo) {
Review Comment:
Change it similarly as `OmBucketInfo`, i.e. use one method to
include/exclude large fields.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ReconOMDBDefinition.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.codec;
+
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.BUCKET_TABLE;
+import static
org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_DIR_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_KEY_TABLE;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.hdds.utils.db.StringCodec;
+import org.apache.hadoop.ozone.ClientVersion;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.codec.OMDBDefinition;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Recon-specific OM RocksDB schema definition.
+ *
+ * <p>
+ * This is a specialized version of {@code
org.apache.hadoop.ozone.om.codec.OMDBDefinition},
+ * used by the Recon service (the Ozone analytics layer). It uses lightweight,
+ * “partial” Protobuf codecs for faster deserialization and to skip loading
+ * ACL or permission details unnecessary for analytic workloads.
+ * </p>
+ *
+ * <p><b>Included Column Families:</b></p>
+ * <ul>
+ * <li>BUCKET_TABLE_DEF: maps volume/bucket → {@link OmBucketInfo}</li>
+ * <li>KEY_TABLE_DEF, OPEN_KEY_TABLE_DEF: map volume/bucket/key* → {@link
OmKeyInfo}</li>
+ * <li>DELETED_TABLE_DEF: maps deleted keys → {@link RepeatedOmKeyInfo}</li>
+ * <li>FILE_TABLE_DEF, OPEN_FILE_TABLE_DEF, DIRECTORY_TABLE_DEF,
DELETED_DIR_TABLE_DEF:
+ * FSO layout support → {@link OmDirectoryInfo}, {@link OmKeyInfo}</li>
+ * <li>… and all other column families imported from the canonical OM
definition, such as:
+ * DELEGATION_TOKEN_TABLE_DEF, META_TABLE_DEF, S3/tenant/snapshot
tables, etc.</li>
+ * </ul>
+ *
+ * <p>
+ * The column families are initialized via {@link DBColumnFamilyDefinition}
and passed
+ * into the parent {@link DBDefinition.WithMap} to initialize Recon's RocksDB
instance.
+ * </p>
+ *
+ * <p><b>Codecs:</b></p>
+ * <ul>
+ * <li>Custom {@link DelegatedCodec} are used to transform partial protobuf
messages into OM helper objects.</li>
+ * <li>Examples include {@link OmBucketInfo#getCodec}-based code reuse for
Om-only read path.</li>
+ * </ul>
+ *
+ * <p><b>Singleton Access:</b></p>
+ * This class implements the singleton pattern:
+ * <pre>
+ * ReconOMDBDefinition def = ReconOMDBDefinition.get();
+ * </pre>
+ *
+ */
+public final class ReconOMDBDefinition extends DBDefinition.WithMap {
+
+ //---------------------------------------------------------------------------
+ // Volume, Bucket, Prefix and Transaction Tables:
+ private static final Codec<OmBucketInfo> CUSTOM_CODEC_FOR_BUCKET_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.BucketInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmBucketInfoFromProtobuf,
+ null,
+ OmBucketInfo.class);
+ /** bucketTable: /volume/bucket :- BucketInfo. */
+ public static final DBColumnFamilyDefinition<String, OmBucketInfo>
BUCKET_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(BUCKET_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_BUCKET_TABLE);
+
+ public static final Codec<OmKeyInfo> CUSTOM_CODEC_FOR_KEY_TABLE = new
DelegatedCodec<>(
+ Proto2Codec.get(OzoneManagerProtocolProtos.KeyInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmKeyInfoFromProtobuf,
+ k -> k.getProtobuf(true, ClientVersion.CURRENT_VERSION),
+ OmKeyInfo.class);
+ // Object Store (OBS) Tables:
+ /** keyTable: /volume/bucket/key :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo> KEY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(KEY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ private static final Codec<RepeatedOmKeyInfo> CUSTOM_CODEC_FOR_DELETED_TABLE
= new DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.RepeatedKeyInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getRepeatedOmKeyInfoFromProto,
+ k -> k.getProto(true, ClientVersion.CURRENT_VERSION),
+ RepeatedOmKeyInfo.class);
+
+ /** deletedTable: /volume/bucket/key :- RepeatedKeyInfo. */
+ public static final DBColumnFamilyDefinition<String, RepeatedOmKeyInfo>
DELETED_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DELETED_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_DELETED_TABLE);
+
+ /** openKeyTable: /volume/bucket/key/id :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
OPEN_KEY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(OPEN_KEY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ //---------------------------------------------------------------------------
+ // File System Optimized (FSO) Tables:
+ /** fileTable: /volumeId/bucketId/parentId/fileName :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
FILE_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(FILE_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ /** openFileTable: /volumeId/bucketId/parentId/fileName/id :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
OPEN_FILE_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(OPEN_FILE_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ public static final Codec<OmDirectoryInfo> CUSTOM_CODEC_FOR_DIR_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.DirectoryInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmDirInfoFromProtobuf,
+ null,
+ OmDirectoryInfo.class);
+ /** directoryTable: /volumeId/bucketId/parentId/dirName :- DirInfo. */
+ public static final DBColumnFamilyDefinition<String, OmDirectoryInfo>
DIRECTORY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DIRECTORY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_DIR_TABLE);
+
+ /** deletedDirectoryTable: /volumeId/bucketId/parentId/dirName/objectId :-
KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
DELETED_DIR_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DELETED_DIR_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ // Merged column family map: OM base + Recon overrides
+ public static final Map<String, DBColumnFamilyDefinition<?, ?>>
COLUMN_FAMILIES;
+
+ //---------------------------------------------------------------------------
+ static {
+ Map<String, DBColumnFamilyDefinition<?, ?>> merged =
+ new HashMap<>(OMDBDefinition.get().getMap());
+ merged.putAll(buildOverrides());
+ COLUMN_FAMILIES = Collections.unmodifiableMap(merged);
+ }
+
+ private static final ReconOMDBDefinition INSTANCE = new
ReconOMDBDefinition();
+
+ /**
+ * Parses BucketInfo protobuf and creates OmBucketInfo without deserializing
ACL list.
+ * @param bucketInfo
+ * @return instance of OmBucketInfo
+ */
+ public static OmBucketInfo
getOmBucketInfoFromProtobuf(OzoneManagerProtocolProtos.BucketInfo bucketInfo) {
+ return OmBucketInfo.newBuilderFromProtobufPartial(bucketInfo).build();
+ }
+
+ //---------------------------------------------------------------------------
+ public static OmKeyInfo
getOmKeyInfoFromProtobuf(OzoneManagerProtocolProtos.KeyInfo keyInfo) {
+ return OmKeyInfo.newBuilderFromProtobufPartial(keyInfo).build();
+ }
+
+ public static RepeatedOmKeyInfo getRepeatedOmKeyInfoFromProto(
+ OzoneManagerProtocolProtos.RepeatedKeyInfo repeatedKeyInfo) {
+ List<OmKeyInfo> list = new ArrayList<>();
+ for (OzoneManagerProtocolProtos.KeyInfo k :
repeatedKeyInfo.getKeyInfoList()) {
+ list.add(ReconOMDBDefinition.getOmKeyInfoFromProtobuf(k));
+ }
+ return new RepeatedOmKeyInfo.Builder().setOmKeyInfos(list).build();
+ }
+
+ /**
+ * Parses DirectoryInfo protobuf and creates OmPrefixInfo.
+ * @param dirInfo
+ * @return instance of OmDirectoryInfo
+ */
Review Comment:
This javadoce does not provide any useful information and generate "tag
description is missing" warning. Please remove it or rewrite it.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ReconOMDBDefinition.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.codec;
+
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.BUCKET_TABLE;
+import static
org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_DIR_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_KEY_TABLE;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.hdds.utils.db.StringCodec;
+import org.apache.hadoop.ozone.ClientVersion;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.codec.OMDBDefinition;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Recon-specific OM RocksDB schema definition.
+ *
+ * <p>
+ * This is a specialized version of {@code
org.apache.hadoop.ozone.om.codec.OMDBDefinition},
+ * used by the Recon service (the Ozone analytics layer). It uses lightweight,
+ * “partial” Protobuf codecs for faster deserialization and to skip loading
+ * ACL or permission details unnecessary for analytic workloads.
+ * </p>
+ *
+ * <p><b>Included Column Families:</b></p>
+ * <ul>
+ * <li>BUCKET_TABLE_DEF: maps volume/bucket → {@link OmBucketInfo}</li>
+ * <li>KEY_TABLE_DEF, OPEN_KEY_TABLE_DEF: map volume/bucket/key* → {@link
OmKeyInfo}</li>
+ * <li>DELETED_TABLE_DEF: maps deleted keys → {@link RepeatedOmKeyInfo}</li>
+ * <li>FILE_TABLE_DEF, OPEN_FILE_TABLE_DEF, DIRECTORY_TABLE_DEF,
DELETED_DIR_TABLE_DEF:
+ * FSO layout support → {@link OmDirectoryInfo}, {@link OmKeyInfo}</li>
+ * <li>… and all other column families imported from the canonical OM
definition, such as:
+ * DELEGATION_TOKEN_TABLE_DEF, META_TABLE_DEF, S3/tenant/snapshot
tables, etc.</li>
+ * </ul>
+ *
+ * <p>
+ * The column families are initialized via {@link DBColumnFamilyDefinition}
and passed
+ * into the parent {@link DBDefinition.WithMap} to initialize Recon's RocksDB
instance.
+ * </p>
+ *
+ * <p><b>Codecs:</b></p>
+ * <ul>
+ * <li>Custom {@link DelegatedCodec} are used to transform partial protobuf
messages into OM helper objects.</li>
+ * <li>Examples include {@link OmBucketInfo#getCodec}-based code reuse for
Om-only read path.</li>
+ * </ul>
+ *
+ * <p><b>Singleton Access:</b></p>
+ * This class implements the singleton pattern:
+ * <pre>
+ * ReconOMDBDefinition def = ReconOMDBDefinition.get();
+ * </pre>
+ *
+ */
+public final class ReconOMDBDefinition extends DBDefinition.WithMap {
+
+ //---------------------------------------------------------------------------
+ // Volume, Bucket, Prefix and Transaction Tables:
+ private static final Codec<OmBucketInfo> CUSTOM_CODEC_FOR_BUCKET_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.BucketInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmBucketInfoFromProtobuf,
+ null,
+ OmBucketInfo.class);
+ /** bucketTable: /volume/bucket :- BucketInfo. */
+ public static final DBColumnFamilyDefinition<String, OmBucketInfo>
BUCKET_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(BUCKET_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_BUCKET_TABLE);
+
+ public static final Codec<OmKeyInfo> CUSTOM_CODEC_FOR_KEY_TABLE = new
DelegatedCodec<>(
+ Proto2Codec.get(OzoneManagerProtocolProtos.KeyInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmKeyInfoFromProtobuf,
+ k -> k.getProtobuf(true, ClientVersion.CURRENT_VERSION),
+ OmKeyInfo.class);
+ // Object Store (OBS) Tables:
+ /** keyTable: /volume/bucket/key :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo> KEY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(KEY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ private static final Codec<RepeatedOmKeyInfo> CUSTOM_CODEC_FOR_DELETED_TABLE
= new DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.RepeatedKeyInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getRepeatedOmKeyInfoFromProto,
+ k -> k.getProto(true, ClientVersion.CURRENT_VERSION),
+ RepeatedOmKeyInfo.class);
+
+ /** deletedTable: /volume/bucket/key :- RepeatedKeyInfo. */
+ public static final DBColumnFamilyDefinition<String, RepeatedOmKeyInfo>
DELETED_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DELETED_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_DELETED_TABLE);
+
+ /** openKeyTable: /volume/bucket/key/id :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
OPEN_KEY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(OPEN_KEY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ //---------------------------------------------------------------------------
+ // File System Optimized (FSO) Tables:
+ /** fileTable: /volumeId/bucketId/parentId/fileName :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
FILE_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(FILE_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ /** openFileTable: /volumeId/bucketId/parentId/fileName/id :- KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
OPEN_FILE_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(OPEN_FILE_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ public static final Codec<OmDirectoryInfo> CUSTOM_CODEC_FOR_DIR_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.DirectoryInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmDirInfoFromProtobuf,
+ null,
+ OmDirectoryInfo.class);
+ /** directoryTable: /volumeId/bucketId/parentId/dirName :- DirInfo. */
+ public static final DBColumnFamilyDefinition<String, OmDirectoryInfo>
DIRECTORY_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DIRECTORY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_DIR_TABLE);
+
+ /** deletedDirectoryTable: /volumeId/bucketId/parentId/dirName/objectId :-
KeyInfo. */
+ public static final DBColumnFamilyDefinition<String, OmKeyInfo>
DELETED_DIR_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(DELETED_DIR_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_KEY_TABLE);
+
+ // Merged column family map: OM base + Recon overrides
+ public static final Map<String, DBColumnFamilyDefinition<?, ?>>
COLUMN_FAMILIES;
+
+ //---------------------------------------------------------------------------
+ static {
+ Map<String, DBColumnFamilyDefinition<?, ?>> merged =
+ new HashMap<>(OMDBDefinition.get().getMap());
+ merged.putAll(buildOverrides());
+ COLUMN_FAMILIES = Collections.unmodifiableMap(merged);
+ }
+
+ private static final ReconOMDBDefinition INSTANCE = new
ReconOMDBDefinition();
+
+ /**
+ * Parses BucketInfo protobuf and creates OmBucketInfo without deserializing
ACL list.
+ * @param bucketInfo
+ * @return instance of OmBucketInfo
+ */
Review Comment:
This javadoce does not provide any useful information and generate "tag
description is missing" warning. Please remove it or rewrite it.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/NSSummaryTaskWithFSO.java:
##########
@@ -183,8 +187,8 @@ public boolean reprocessWithFSO(OMMetadataManager
omMetadataManager) {
Map<Long, NSSummary> nsSummaryMap = new HashMap<>();
try {
- Table<String, OmDirectoryInfo> dirTable =
- omMetadataManager.getDirectoryTable();
+ Table<String, OmDirectoryInfo> dirTable = omMetadataManager.getStore()
+ .getTable(DIRECTORY_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_DIR_TABLE, TableCache.CacheType.NO_CACHE);
Review Comment:
Use DEF:
```java
final Table<String, OmDirectoryInfo> dirTable =
ReconOMDBDefinition.DIRECTORY_TABLE_DEF
.getTable(omMetadataManager.getStore());
```
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ReconOMDBDefinition.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.codec;
+
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.BUCKET_TABLE;
+import static
org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_DIR_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_KEY_TABLE;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.hdds.utils.db.StringCodec;
+import org.apache.hadoop.ozone.ClientVersion;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.codec.OMDBDefinition;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Recon-specific OM RocksDB schema definition.
+ *
+ * <p>
+ * This is a specialized version of {@code
org.apache.hadoop.ozone.om.codec.OMDBDefinition},
+ * used by the Recon service (the Ozone analytics layer). It uses lightweight,
+ * “partial” Protobuf codecs for faster deserialization and to skip loading
+ * ACL or permission details unnecessary for analytic workloads.
+ * </p>
+ *
+ * <p><b>Included Column Families:</b></p>
+ * <ul>
+ * <li>BUCKET_TABLE_DEF: maps volume/bucket → {@link OmBucketInfo}</li>
+ * <li>KEY_TABLE_DEF, OPEN_KEY_TABLE_DEF: map volume/bucket/key* → {@link
OmKeyInfo}</li>
+ * <li>DELETED_TABLE_DEF: maps deleted keys → {@link RepeatedOmKeyInfo}</li>
+ * <li>FILE_TABLE_DEF, OPEN_FILE_TABLE_DEF, DIRECTORY_TABLE_DEF,
DELETED_DIR_TABLE_DEF:
+ * FSO layout support → {@link OmDirectoryInfo}, {@link OmKeyInfo}</li>
+ * <li>… and all other column families imported from the canonical OM
definition, such as:
+ * DELEGATION_TOKEN_TABLE_DEF, META_TABLE_DEF, S3/tenant/snapshot
tables, etc.</li>
+ * </ul>
+ *
+ * <p>
+ * The column families are initialized via {@link DBColumnFamilyDefinition}
and passed
+ * into the parent {@link DBDefinition.WithMap} to initialize Recon's RocksDB
instance.
+ * </p>
+ *
+ * <p><b>Codecs:</b></p>
+ * <ul>
+ * <li>Custom {@link DelegatedCodec} are used to transform partial protobuf
messages into OM helper objects.</li>
+ * <li>Examples include {@link OmBucketInfo#getCodec}-based code reuse for
Om-only read path.</li>
+ * </ul>
+ *
+ * <p><b>Singleton Access:</b></p>
+ * This class implements the singleton pattern:
+ * <pre>
+ * ReconOMDBDefinition def = ReconOMDBDefinition.get();
+ * </pre>
+ *
+ */
+public final class ReconOMDBDefinition extends DBDefinition.WithMap {
+
+ //---------------------------------------------------------------------------
+ // Volume, Bucket, Prefix and Transaction Tables:
+ private static final Codec<OmBucketInfo> CUSTOM_CODEC_FOR_BUCKET_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.BucketInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmBucketInfoFromProtobuf,
+ null,
+ OmBucketInfo.class);
+ /** bucketTable: /volume/bucket :- BucketInfo. */
+ public static final DBColumnFamilyDefinition<String, OmBucketInfo>
BUCKET_TABLE_DEF
+ = new DBColumnFamilyDefinition<>(BUCKET_TABLE, StringCodec.get(),
CUSTOM_CODEC_FOR_BUCKET_TABLE);
+
+ public static final Codec<OmKeyInfo> CUSTOM_CODEC_FOR_KEY_TABLE = new
DelegatedCodec<>(
Review Comment:
It is not only for KEY_TABLE. Let's rename it to `CUSTOM_OM_KEY_INFO_CODEC`.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/codec/ReconOMDBDefinition.java:
##########
@@ -0,0 +1,226 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.ozone.recon.codec;
+
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.BUCKET_TABLE;
+import static
org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_DIR_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DELETED_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.DIRECTORY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.KEY_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_FILE_TABLE;
+import static org.apache.hadoop.ozone.om.codec.OMDBDefinition.OPEN_KEY_TABLE;
+
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import org.apache.hadoop.hdds.utils.db.Codec;
+import org.apache.hadoop.hdds.utils.db.DBColumnFamilyDefinition;
+import org.apache.hadoop.hdds.utils.db.DBDefinition;
+import org.apache.hadoop.hdds.utils.db.DelegatedCodec;
+import org.apache.hadoop.hdds.utils.db.Proto2Codec;
+import org.apache.hadoop.hdds.utils.db.StringCodec;
+import org.apache.hadoop.ozone.ClientVersion;
+import org.apache.hadoop.ozone.OzoneConsts;
+import org.apache.hadoop.ozone.om.OMConfigKeys;
+import org.apache.hadoop.ozone.om.codec.OMDBDefinition;
+import org.apache.hadoop.ozone.om.helpers.OmBucketInfo;
+import org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo;
+import org.apache.hadoop.ozone.om.helpers.OmKeyInfo;
+import org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo;
+import org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos;
+
+/**
+ * Recon-specific OM RocksDB schema definition.
+ *
+ * <p>
+ * This is a specialized version of {@code
org.apache.hadoop.ozone.om.codec.OMDBDefinition},
+ * used by the Recon service (the Ozone analytics layer). It uses lightweight,
+ * “partial” Protobuf codecs for faster deserialization and to skip loading
+ * ACL or permission details unnecessary for analytic workloads.
+ * </p>
+ *
+ * <p><b>Included Column Families:</b></p>
+ * <ul>
+ * <li>BUCKET_TABLE_DEF: maps volume/bucket → {@link OmBucketInfo}</li>
+ * <li>KEY_TABLE_DEF, OPEN_KEY_TABLE_DEF: map volume/bucket/key* → {@link
OmKeyInfo}</li>
+ * <li>DELETED_TABLE_DEF: maps deleted keys → {@link RepeatedOmKeyInfo}</li>
+ * <li>FILE_TABLE_DEF, OPEN_FILE_TABLE_DEF, DIRECTORY_TABLE_DEF,
DELETED_DIR_TABLE_DEF:
+ * FSO layout support → {@link OmDirectoryInfo}, {@link OmKeyInfo}</li>
+ * <li>… and all other column families imported from the canonical OM
definition, such as:
+ * DELEGATION_TOKEN_TABLE_DEF, META_TABLE_DEF, S3/tenant/snapshot
tables, etc.</li>
+ * </ul>
+ *
+ * <p>
+ * The column families are initialized via {@link DBColumnFamilyDefinition}
and passed
+ * into the parent {@link DBDefinition.WithMap} to initialize Recon's RocksDB
instance.
+ * </p>
+ *
+ * <p><b>Codecs:</b></p>
+ * <ul>
+ * <li>Custom {@link DelegatedCodec} are used to transform partial protobuf
messages into OM helper objects.</li>
+ * <li>Examples include {@link OmBucketInfo#getCodec}-based code reuse for
Om-only read path.</li>
+ * </ul>
+ *
+ * <p><b>Singleton Access:</b></p>
+ * This class implements the singleton pattern:
+ * <pre>
+ * ReconOMDBDefinition def = ReconOMDBDefinition.get();
+ * </pre>
+ *
+ */
+public final class ReconOMDBDefinition extends DBDefinition.WithMap {
+
+ //---------------------------------------------------------------------------
+ // Volume, Bucket, Prefix and Transaction Tables:
+ private static final Codec<OmBucketInfo> CUSTOM_CODEC_FOR_BUCKET_TABLE = new
DelegatedCodec<>(
+
Proto2Codec.get(OzoneManagerProtocolProtos.BucketInfo.getDefaultInstance()),
+ ReconOMDBDefinition::getOmBucketInfoFromProtobuf,
+ null,
Review Comment:
Use `OmBucketInfo::getProtobuf` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
