Jackie-Jiang commented on code in PR #20113:
URL: https://github.com/apache/pinot/pull/20113#discussion_r4532008596
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/utils/CrcUtils.java:
##########
@@ -42,30 +43,58 @@ private CrcUtils() {
private static final Logger LOGGER = LoggerFactory.getLogger(CrcUtils.class);
private static final int BUFFER_SIZE = 65536;
private static final String CRC_FILE_EXTENSTION = ".crc";
- private static final List<String> DATA_FILE_EXTENSIONS =
Arrays.asList(".fwd", ".dict");
+ private static final List<String> DATA_FILE_EXTENSIONS_V1 =
Arrays.asList(".fwd", ".dict");
+ // Null vectors count as data in version 2, because a null is stored as the
default value in the forward index
+ private static final List<String> DATA_FILE_EXTENSIONS_V2 =
+ Arrays.asList(".fwd", ".dict",
V1Constants.Indexes.NULLVALUE_VECTOR_FILE_EXTENSION);
+ private static volatile int _defaultDataCrcVersion =
V1Constants.DataCrc.VERSION_1;
+
+ /// Sets the data CRC version used for segments built in this JVM.
+ public static void setDefaultDataCrcVersion(int dataCrcVersion) {
+ _defaultDataCrcVersion = validateDataCrcVersion(dataCrcVersion);
+ }
+
+ public static int getDefaultDataCrcVersion() {
+ return _defaultDataCrcVersion;
+ }
public static long computeCrc(File indexDir) throws IOException {
List<File> allNormalFiles = new ArrayList<>();
- collectFiles(indexDir, allNormalFiles, false);
+ collectFiles(indexDir, allNormalFiles, null);
Collections.sort(allNormalFiles);
return crcForFiles(allNormalFiles);
}
public static long computeDataCrc(File indexDir) throws IOException {
+ return computeDataCrc(indexDir, V1Constants.DataCrc.VERSION_1);
+ }
+
+ /// Computes the data CRC with the given version. Version 2 equals version 1
when the segment has no null vectors.
+ public static long computeDataCrc(File indexDir, int dataCrcVersion) throws
IOException {
+ List<String> dataFileExtensions =
Review Comment:
Consider encoding the version in the existing data CRC long. Adler32 uses
only the lower 32 bits, so we can make version 0 the default/legacy algorithm
and version 1 the algorithm including null vectors:
```java
return ((long) version << 32) | checksum;
```
Version 0 preserves every existing checksum value. Version 1 remains
positive and cannot match a legacy checksum. The existing long equality checks
then compare both version and checksum automatically, including on old servers.
The OSS metadata, ZK, upload-response, and valid-doc-ID paths I checked
preserve the full long value.
This would remove the appended integer in creation.meta, the separate
metadata field/SPI getter/JSON property, and the need to propagate a separate
version through ZK and upload responses. Keep supported-version validation and
leave negative unavailable sentinels unencoded.
It also closes the mixed-version gap in the current implementation: a legacy
copy containing NULL can have the same data CRC as a new-algorithm copy
containing the corresponding stored default. The separate version currently
does not participate in hasSameCRC, so the claim that these cases always fall
back to full CRC is not correct.
The deliberate tradeoff is that cross-version data CRCs also differ for
segments without nulls. Full CRC equality should remain sufficient; otherwise
the replica downloads the committed copy. Please cover the encoding and
round-trip values above 0xffffffff, plus mixed-version comparisons with equal
and unequal full CRCs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]