gianm commented on code in PR #17691:
URL: https://github.com/apache/druid/pull/17691#discussion_r1939795608
##########
processing/src/main/java/org/apache/druid/segment/data/GenericIndexedWriter.java:
##########
@@ -72,18 +73,31 @@ public class GenericIndexedWriter<T> implements
DictionaryWriter<T>
.writeByteArray(x -> x.fileNameByteArray);
+ /**
+ * Creates a new writer that accepts byte buffers and compresses them.
+ *
+ * @param segmentWriteOutMedium supplier of temporary files
+ * @param filenameBase base filename to be used for secondary
files, if multiple files are needed
+ * @param compressionStrategy compression strategy to apply
+ * @param bufferSize size of the buffers that will be passed in
+ * @param fileSizeLimit limit for files created by the writer. In
production code, this should always be
+ * {@link GenericIndexedWriter#MAX_FILE_SIZE}.
The parameter is exposed only for testing.
+ * @param closer closer to attach temporary compression
buffers to
+ */
public static GenericIndexedWriter<ByteBuffer> ofCompressedByteBuffers(
final SegmentWriteOutMedium segmentWriteOutMedium,
final String filenameBase,
final CompressionStrategy compressionStrategy,
final int bufferSize,
+ final int fileSizeLimit,
final Closer closer
)
{
GenericIndexedWriter<ByteBuffer> writer = new GenericIndexedWriter<>(
segmentWriteOutMedium,
filenameBase,
- compressedByteBuffersWriteObjectStrategy(compressionStrategy,
bufferSize, closer)
+ compressedByteBuffersWriteObjectStrategy(compressionStrategy,
bufferSize, closer),
+ fileSizeLimit
Review Comment:
The crux of the fix was passing in a `SmooshedFileMapper` to
`GenericIndexed#read` in various call sites. The new `fileSizeLimit` parameter
is really just there for testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
