nsivabalan commented on code in PR #13952:
URL: https://github.com/apache/hudi/pull/13952#discussion_r2366036229
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -86,21 +84,8 @@ public Pair<HoodieFileGroupRecordBuffer<T>, List<String>>
getRecordBuffer(Hoodie
deleteContext.withReaderSchema(recordSchema);
while (recordIterator.hasNext()) {
HoodieRecord<T> hoodieRecord = recordIterator.next();
- T data = recordContext.extractDataFromRecord(hoodieRecord, recordSchema,
props);
try {
- // we use -U operation to represent the record should be ignored
during updating index.
- HoodieOperation hoodieOperation = hoodieRecord.getIgnoreIndexUpdate()
? HoodieOperation.UPDATE_BEFORE : hoodieRecord.getOperation();
- BufferedRecord<T> bufferedRecord;
- if (data == null) {
- DeleteRecord deleteRecord =
DeleteRecord.create(hoodieRecord.getKey(),
hoodieRecord.getOrderingValue(recordSchema, props, orderingFieldsArray));
- bufferedRecord = BufferedRecords.fromDeleteRecord(deleteRecord,
recordContext, hoodieOperation);
- } else {
- // HoodieRecord#isDelete does not check if a record is a DELETE
marked by a custom delete marker,
- // so we use recordContext#isDeleteRecord here if the data field is
not null.
- boolean isDelete = recordContext.isDeleteRecord(data, deleteContext);
- bufferedRecord = BufferedRecords.fromEngineRecord(data,
hoodieRecord.getRecordKey(), recordSchema, recordContext, orderingFieldNames,
- BufferedRecords.inferOperation(isDelete, hoodieOperation));
- }
+ BufferedRecord<T> bufferedRecord =
BufferedRecords.fromHoodieRecord(hoodieRecord, recordSchema, recordContext,
props, orderingFieldsArray);
Review Comment:
we do have ordering value in HoodieRecord, but here looks like we are again
trying to fetch from data (within
`BufferedRecords.fromHoodieRecord(hoodieRecord, recordSchema, recordContext,
props, orderingFieldsArray)` )
##########
hudi-common/src/main/java/org/apache/hudi/common/model/SerializableIndexedRecord.java:
##########
@@ -46,6 +46,7 @@ public class SerializableIndexedRecord implements
GenericRecord, KryoSerializabl
private static final long serialVersionUID = 1L;
private IndexedRecord record;
+ private Schema schema;
Review Comment:
is this set just before merging and writing to new file?
if not, can we fix `HoodieAvroIndexedRecord` only to pass in schema while
calling `put` or `get` methods. so that we can take in additional argument as
schema and avoid storing it here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
