hangc0276 commented on a change in pull request #11490:
URL: https://github.com/apache/pulsar/pull/11490#discussion_r679214179
##########
File path:
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
##########
@@ -47,11 +53,27 @@
private final MetadataStore store;
private final OrderedExecutor executor;
+ public static final short MAGIC_MANAGED_LEDGER_INFO_METADATA = 0x0b9c;
+ private CompressionType compressionType = null;
Review comment:
initiate to be `None` instead of `null`?
##########
File path: managed-ledger/src/main/proto/MLDataFormats.proto
##########
@@ -124,3 +124,8 @@ message ManagedCursorInfo {
// Store which index in the batch message has been deleted
repeated BatchedEntryDeletionIndexInfo batchedEntryDeletionIndexInfo = 7;
}
+
+message ManagedLedgerInfoMetadata {
+ required string compressionType = 1;
Review comment:
Use compression enum type instead of String？
##########
File path:
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
##########
@@ -264,4 +285,70 @@ private static MetaStoreException getException(Throwable
t) {
return new MetaStoreException(t);
}
}
+
+ /**
+ * Compress ManagedLedgerInfo data.
+ *
+ * compression data structure
+ * [MAGIC_NUMBER](2) + [METADATA_SIZE](4) + [METADATA_PAYLOAD] +
[MANAGED_LEDGER_INFO_PAYLOAD]
+ */
+ public byte[] compressLedgerInfo(ManagedLedgerInfo managedLedgerInfo) {
+ if (compressionType == null ||
compressionType.equals(CompressionType.NONE)) {
+ return managedLedgerInfo.toByteArray();
+ }
+ byte[] originalBytes = managedLedgerInfo.toByteArray();
+ MLDataFormats.ManagedLedgerInfoMetadata mlInfoMetadata =
MLDataFormats.ManagedLedgerInfoMetadata.newBuilder()
+ .setCompressionType(compressionType.name())
+ .setUnpressedSize(originalBytes.length)
+ .build();
+ ByteBuf metadataByteBuf = PulsarByteBufAllocator.DEFAULT.buffer(
+ mlInfoMetadata.getSerializedSize() + 6,
mlInfoMetadata.getSerializedSize() + 6);
+ metadataByteBuf.writeShort(MAGIC_MANAGED_LEDGER_INFO_METADATA);
+ metadataByteBuf.writeInt(mlInfoMetadata.getSerializedSize());
+ metadataByteBuf.writeBytes(mlInfoMetadata.toByteArray());
+
+ ByteBuf originalByteBuf =
PulsarByteBufAllocator.DEFAULT.buffer(originalBytes.length,
originalBytes.length);
+ originalByteBuf.writeBytes(originalBytes);
+ ByteBuf encodeByteBuf =
CompressionCodecProvider.getCompressionCodec(compressionType).encode(originalByteBuf);
+
+ CompositeByteBuf compositeByteBuf =
PulsarByteBufAllocator.DEFAULT.compositeBuffer();
+ compositeByteBuf.addComponent(true, metadataByteBuf);
+ compositeByteBuf.addComponent(true, encodeByteBuf);
+
+ byte[] dataBytes = new byte[compositeByteBuf.readableBytes()];
+ compositeByteBuf.readBytes(dataBytes);
+ return dataBytes;
+ }
+
+ public ManagedLedgerInfo parseManagedLedgerInfo(byte[] data) throws
InvalidProtocolBufferException {
+ ByteBuf byteBuf = PulsarByteBufAllocator.DEFAULT.buffer(data.length,
data.length);
+ byteBuf.writeBytes(data);
+ if (byteBuf.readableBytes() > 0 && byteBuf.readShort() ==
MAGIC_MANAGED_LEDGER_INFO_METADATA) {
+ try {
+ int metadataSize = byteBuf.readInt();
+ byte[] metadataBytes = new byte[metadataSize];
+ byteBuf.readBytes(metadataBytes);
+ MLDataFormats.ManagedLedgerInfoMetadata metadata =
+
MLDataFormats.ManagedLedgerInfoMetadata.parseFrom(metadataBytes);
+
+ long unpressedSize = metadata.getUnpressedSize();
+ ByteBuf decodeByteBuf =
CompressionCodecProvider.getCompressionCodec(
Review comment:
```
/**
* Decompress a buffer.
*
* <p>The buffer needs to have been compressed with the matching Encoder.
*
* @param encoded
* the compressed content
* @param uncompressedSize
* the size of the original content
* @return a ByteBuf with the compressed content. The buffer needs to be
released by the receiver
* @throws IOException
* if the decompression fails
*/
ByteBuf decode(ByteBuf encoded, int uncompressedSize) throws IOException;
```
we need to release the ByteBuf
##########
File path:
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/ManagedLedgerFactoryConfig.java
##########
@@ -75,4 +75,9 @@
* cluster name for prometheus stats
*/
private String clusterName;
+
+ /**
+ * ManagedLedgerInfo compression type. If the compression type is null or
invalid, don't compress data.
+ */
+ private String managedLedgerInfoCompressionType = null;
Review comment:
We'd better use enum type to strict compression type info specific type
instead of string. I'm not sure whether it's easy to implement.
##########
File path:
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
##########
@@ -264,4 +285,70 @@ private static MetaStoreException getException(Throwable
t) {
return new MetaStoreException(t);
}
}
+
+ /**
+ * Compress ManagedLedgerInfo data.
+ *
+ * compression data structure
+ * [MAGIC_NUMBER](2) + [METADATA_SIZE](4) + [METADATA_PAYLOAD] +
[MANAGED_LEDGER_INFO_PAYLOAD]
+ */
+ public byte[] compressLedgerInfo(ManagedLedgerInfo managedLedgerInfo) {
+ if (compressionType == null ||
compressionType.equals(CompressionType.NONE)) {
+ return managedLedgerInfo.toByteArray();
+ }
+ byte[] originalBytes = managedLedgerInfo.toByteArray();
+ MLDataFormats.ManagedLedgerInfoMetadata mlInfoMetadata =
MLDataFormats.ManagedLedgerInfoMetadata.newBuilder()
+ .setCompressionType(compressionType.name())
+ .setUnpressedSize(originalBytes.length)
+ .build();
+ ByteBuf metadataByteBuf = PulsarByteBufAllocator.DEFAULT.buffer(
+ mlInfoMetadata.getSerializedSize() + 6,
mlInfoMetadata.getSerializedSize() + 6);
+ metadataByteBuf.writeShort(MAGIC_MANAGED_LEDGER_INFO_METADATA);
+ metadataByteBuf.writeInt(mlInfoMetadata.getSerializedSize());
+ metadataByteBuf.writeBytes(mlInfoMetadata.toByteArray());
+
+ ByteBuf originalByteBuf =
PulsarByteBufAllocator.DEFAULT.buffer(originalBytes.length,
originalBytes.length);
+ originalByteBuf.writeBytes(originalBytes);
Review comment:
good idea.
##########
File path:
managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/MetaStoreImpl.java
##########
@@ -264,4 +285,70 @@ private static MetaStoreException getException(Throwable
t) {
return new MetaStoreException(t);
}
}
+
+ /**
+ * Compress ManagedLedgerInfo data.
+ *
+ * compression data structure
+ * [MAGIC_NUMBER](2) + [METADATA_SIZE](4) + [METADATA_PAYLOAD] +
[MANAGED_LEDGER_INFO_PAYLOAD]
+ */
+ public byte[] compressLedgerInfo(ManagedLedgerInfo managedLedgerInfo) {
+ if (compressionType == null ||
compressionType.equals(CompressionType.NONE)) {
+ return managedLedgerInfo.toByteArray();
+ }
+ byte[] originalBytes = managedLedgerInfo.toByteArray();
+ MLDataFormats.ManagedLedgerInfoMetadata mlInfoMetadata =
MLDataFormats.ManagedLedgerInfoMetadata.newBuilder()
+ .setCompressionType(compressionType.name())
+ .setUnpressedSize(originalBytes.length)
+ .build();
+ ByteBuf metadataByteBuf = PulsarByteBufAllocator.DEFAULT.buffer(
+ mlInfoMetadata.getSerializedSize() + 6,
mlInfoMetadata.getSerializedSize() + 6);
+ metadataByteBuf.writeShort(MAGIC_MANAGED_LEDGER_INFO_METADATA);
+ metadataByteBuf.writeInt(mlInfoMetadata.getSerializedSize());
+ metadataByteBuf.writeBytes(mlInfoMetadata.toByteArray());
+
+ ByteBuf originalByteBuf =
PulsarByteBufAllocator.DEFAULT.buffer(originalBytes.length,
originalBytes.length);
+ originalByteBuf.writeBytes(originalBytes);
+ ByteBuf encodeByteBuf =
CompressionCodecProvider.getCompressionCodec(compressionType).encode(originalByteBuf);
Review comment:
```
/**
* Compress a buffer.
*
* @param raw
* a buffer with the uncompressed content. The reader/writer
indexes will not be modified
* @return a new buffer with the compressed content. The buffer needs to
be released by the receiver
*/
ByteBuf encode(ByteBuf raw);
```
Need to be released.
##########
File path:
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/ServiceConfiguration.java
##########
@@ -1601,6 +1601,11 @@
private String managedLedgerDataReadPriority =
OffloadedReadPriority.TIERED_STORAGE_FIRST
.getValue();
+ @FieldContext(category = CATEGORY_STORAGE_ML,
+ doc = "ManagedLedgerInfo compression type, option values (LZ4,
ZLIB, ZSTD, SNAPPY). \n"
Review comment:
add `NONE` compression value type to doc, and default value set to be
`NONE` instead of `null`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
