prateekm commented on a change in pull request #1079: SAMZA-2250: Support large
job models in standalone.
URL: https://github.com/apache/samza/pull/1079#discussion_r296020405
##########
File path: samza-core/src/main/java/org/apache/samza/zk/ZkMetadataStore.java
##########
@@ -125,7 +145,40 @@ public void close() {
zkClient.close();
}
- private String getZkPathForKey(String key) {
- return String.format("%s/%s", zkBaseDir, key);
+ private String getZkPath(String key, int segmentIndex) {
+ return String.format("%s/%s/%d", zkBaseDir, key, segmentIndex);
+ }
+
+ /**
+ * Computes and returns the crc32 checksum of the input byte array.
+ * @param value the input byte array.
+ * @return the crc32 checksum of the byte array.
+ */
+ private static byte[] getCRCChecksum(byte[] value) {
+ CRC32 crc32 = new CRC32();
+ crc32.update(value);
+ long checksum = crc32.getValue();
+ return Longs.toByteArray(checksum);
+ }
+
+ /**
+ * Splits the input byte array value into independent byte array segments of
1 MB size.
+ * @param value the input byte array to split.
+ * @return the byte array splitted into independent byte array chunks.
+ */
+ private static List<byte[]> chunkMetadataStoreValue(byte[] value) {
+ try {
+ byte[] valueCrcAsBytes = getCRCChecksum(value);
+ value = ArrayUtils.addAll(value, valueCrcAsBytes);
Review comment:
Nitpick: prefer new variable instead of reassigning input params.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
