xiangfu0 commented on code in PR #19898:
URL: https://github.com/apache/pinot/pull/19898#discussion_r4309773381
##########
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/spec/Constants.java:
##########
@@ -18,11 +18,23 @@
*/
package org.apache.pinot.spi.ingestion.batch.spec;
+import java.util.Map;
+
public class Constants {
private Constants() {
}
/// By default Pinot segments are compressed in 'tar.gz' format then pushed
to controller.
public static final String TAR_GZ_FILE_EXT = ".tar.gz";
public static final String METADATA_TAR_GZ_FILE_EXT = ".metadata.tar.gz";
+
+ /// Archive extensions understood by segment upload and metadata extraction,
including existing LZ4 archives.
+ public static final Map<String, String> SEGMENT_TAR_FILE_EXTENSIONS =
+ Map.of("gzip", TAR_GZ_FILE_EXT, "lz4", ".tar.lz4", "zstd", ".tar.zst");
Review Comment:
Replaced codec names and archive extensions with named constants in
`Constants.java`, and reused them in the extension map and job-spec validation.
Fixed in
[b9a5baeb57](https://github.com/apache/pinot/commit/b9a5baeb57fe73eccdbe8fb139a7da43fba1d475);
42 focused JDK 25 tests, precommit checks, and independent Pinot review passed.
##########
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/spec/SegmentGenerationJobSpec.java:
##########
@@ -58,6 +59,9 @@ public class SegmentGenerationJobSpec implements Serializable
{
/// Segment creation job parallelism.
private int _segmentCreationJobParallelism;
+ /// Codec for segment archives: `gzip` (default) or `zstd`. Metadata
sidecars always use gzip.
+ private String _segmentTarCompressionCodec = "gzip";
Review Comment:
Added `DEFAULT_SEGMENT_COMPRESSION_CODEC = SEGMENT_COMPRESSION_CODEC_GZIP`
and used it for both field initialization and the getter fallback. Fixed in
[b9a5baeb57](https://github.com/apache/pinot/commit/b9a5baeb57fe73eccdbe8fb139a7da43fba1d475).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
