vamshikrishnakyatham commented on code in PR #13862:
URL: https://github.com/apache/hudi/pull/13862#discussion_r2331700609
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieAvroDataBlock.java:
##########
@@ -108,24 +113,32 @@ protected ByteArrayOutputStream
serializeRecords(List<HoodieRecord> records, Hoo
// 1. Write out the log block version
output.writeInt(HoodieLogBlock.version);
- // 2. Write total number of records
- output.writeInt(records.size());
-
- // 3. Write the records
+ // 2. Pre-serialize records to handle and get accurate count
Properties props = initProperties(storage.getConf());
+ List<ByteArrayOutputStream> serializedRecords = new ArrayList<>();
for (HoodieRecord<?> s : records) {
try {
// Encode the record into bytes
// Spark Record not support write avro log
ByteArrayOutputStream data = s.getAvroBytes(schema, props);
- // Write the record size
- output.writeInt(data.size());
- // Write the content
- data.writeTo(output);
+ serializedRecords.add(data);
} catch (IOException e) {
throw new HoodieIOException("IOException converting
HoodieAvroDataBlock to bytes", e);
+ } catch (Exception e) {
+ LOG.warn("Skipping record during serialization: {}. This may be due
to concurrent archiving race conditions. "
Review Comment:
yeah, it was difficult from the logs to know which field is exactly causing
this as I couldn't catch that in local test running, I will add explicit lock
providers for the test and revert these back
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
