vinothchandar commented on a change in pull request #4449:
URL: https://github.com/apache/hudi/pull/4449#discussion_r776869570
##########
File path:
hudi-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileReader.java
##########
@@ -62,6 +64,7 @@
// Scanner used to read individual keys. This is cached to prevent the
overhead of opening the scanner for each
// key retrieval.
private HFileScanner keyScanner;
+ private final String keyField = HoodieMetadataPayload.SCHEMA_FIELD_ID_KEY;
Review comment:
same comment
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieHFileDataBlock.java
##########
@@ -162,6 +158,20 @@ protected void createRecordsFromContentBytes() throws
IOException {
return records;
}
+ /**
+ * Serialize the record to byte buffer.
+ *
+ * @param record - Record to serialize
+ * @param schemaKeyField - Key field in the schema
+ * @return Serialized byte buffer for the record
+ */
+ private byte[] serializeRecord(final IndexedRecord record, final
Option<Field> schemaKeyField) {
+ if (schemaKeyField.isPresent()) {
+ record.put(schemaKeyField.get().pos(), "");
Review comment:
StringUtils.EMPTY_STRING or some jvm constant?
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##########
@@ -77,6 +81,8 @@ public HoodieHFileWriter(String instantTime, Path file,
HoodieHFileConfig hfileC
this.file = HoodieWrapperFileSystem.convertToHoodiePath(file, conf);
this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(conf);
this.hfileConfig = hfileConfig;
+ this.schema = schema;
+ this.schemaRecordKeyField =
Option.ofNullable(schema.getField(HoodieMetadataPayload.SCHEMA_FIELD_ID_KEY));
Review comment:
Can't this be passed from the metadata code path that actually
instantiates the writer? Then we need not hardcode anything? What am I missing?
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##########
@@ -77,6 +81,8 @@ public HoodieHFileWriter(String instantTime, Path file,
HoodieHFileConfig hfileC
this.file = HoodieWrapperFileSystem.convertToHoodiePath(file, conf);
this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(conf);
this.hfileConfig = hfileConfig;
+ this.schema = schema;
+ this.schemaRecordKeyField =
Option.ofNullable(schema.getField(HoodieMetadataPayload.SCHEMA_FIELD_ID_KEY));
Review comment:
here we do hfileConfig.getKeyFieldID or name or something?
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##########
@@ -122,7 +128,13 @@ public boolean canWrite() {
@Override
public void writeAvro(String recordKey, IndexedRecord object) throws
IOException {
- byte[] value = HoodieAvroUtils.avroToBytes((GenericRecord)object);
+ byte[] value = HoodieAvroUtils.avroToBytes((GenericRecord) object);
Review comment:
why do the avroToBytes twice? better to check the condition first.?
##########
File path:
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/storage/HoodieHFileWriter.java
##########
@@ -77,6 +81,8 @@ public HoodieHFileWriter(String instantTime, Path file,
HoodieHFileConfig hfileC
this.file = HoodieWrapperFileSystem.convertToHoodiePath(file, conf);
this.fs = (HoodieWrapperFileSystem) this.file.getFileSystem(conf);
this.hfileConfig = hfileConfig;
+ this.schema = schema;
+ this.schemaRecordKeyField =
Option.ofNullable(schema.getField(HoodieMetadataPayload.SCHEMA_FIELD_ID_KEY));
Review comment:
i.e a new property `hoodie.hbase...key.field.name` or something, which
is always set to `HoodieMetadataPayload.SCHEMA_FIELD_ID_KEY`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
