cshuo commented on code in PR #19118:
URL: https://github.com/apache/hudi/pull/19118#discussion_r3526122588
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -103,6 +109,21 @@ public boolean isDataOrDeleteBlock() {
return getBlockType().isDataOrDeleteBlock();
}
+ protected HoodieSchema getSchemaFromHeader() {
Review Comment:
For this path the cache key is the schema string from the log block header
(`String -> HoodieSchema`), while `HoodieAvroSchemaCache` caches an
already-parsed Avro `Schema -> HoodieSchema`. This is also not new caching
logic; it was moved from `HoodieDataBlock` into `HoodieLogBlock` so native
data/delete blocks can share the same header-schema parsing path.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -137,9 +137,9 @@ public DeleteRecord[] getRecordsToDelete() {
/**
* Returns delete records in the same representation used by the file-group
record buffer.
*/
- public <T> List<BufferedRecord<T>> getRecordsToDelete(RecordContext<T>
recordContext) {
+ public <T> List<BufferedRecord<T>> getRecordsToDelete(HoodieReaderContext<T>
readerContext) {
return Arrays.stream(getRecordsToDelete())
- .map(deleteRecord -> BufferedRecords.fromDeleteRecord(deleteRecord,
recordContext))
+ .map(deleteRecord -> BufferedRecords.fromDeleteRecord(deleteRecord,
readerContext.getRecordContext()))
Review Comment:
This signature changed because native delete logs need the full
`HoodieReaderContext` to build a file iterator over the native delete file.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieNativeDeleteBlock.java:
##########
@@ -46,51 +52,76 @@ public class HoodieNativeDeleteBlock extends
HoodieDeleteBlock {
private final HoodieStorage storage;
private final HoodieLogFile logFile;
- private final HoodieReaderContext<?> readerContext;
private final HoodieSchema deleteLogSchema;
private final List<String> orderingFieldNames;
+ private final String partitionPath;
+ private final Properties props;
+ private DeleteRecord[] recordsToDelete;
private List<BufferedRecord<?>> bufferedRecordsToDelete;
public HoodieNativeDeleteBlock(HoodieStorage storage,
HoodieLogFile logFile,
- HoodieReaderContext<?> readerContext,
- HoodieSchema deleteLogSchema,
List<String> orderingFieldNames,
+ String partitionPath,
+ Properties props,
Map<HeaderMetadataType, String> header,
Map<FooterMetadataType, String> footer) {
super(Option.empty(), null, true, getContentLocation(storage, logFile),
header, footer);
this.storage = storage;
this.logFile = logFile;
- this.readerContext = readerContext;
- this.deleteLogSchema = deleteLogSchema;
+ this.deleteLogSchema =
HoodieSchemas.createDeleteLogSchema(getSchemaFromHeader(), orderingFieldNames);
this.orderingFieldNames = orderingFieldNames;
+ this.partitionPath = partitionPath;
+ this.props = props == null ? new Properties() : props;
}
@Override
public DeleteRecord[] getRecordsToDelete() {
- throw new HoodieNotSupportedException("Native delete log files do not
support the legacy DeleteRecord[] API. "
- + "Use getRecordsToDelete(RecordContext) instead. Log file: " +
logFile);
+ if (recordsToDelete == null) {
+ recordsToDelete = readRecordsToDelete();
Review Comment:
This path reconstructs `DeleteRecord` for the legacy scanner/delete-block
API, while the file-group reader merge path uses the `BufferedRecord` delete
representation through `getRecordsToDelete(HoodieReaderContext)`, for e.g.,
hive readers.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/util/CommonClientUtils.java:
##########
@@ -120,6 +130,23 @@ public static HoodieLogBlock.HoodieLogBlockType
getLogBlockType(HoodieWriteConfi
}
}
+ /**
+ * Whether log blocks should be written in the native (v2) log format
(standalone native
+ * files written via {@code HoodieNativeLogFormatWriter}) instead of the
legacy inline
+ * log format. The native format is the default for write version >=
{@link HoodieTableVersion#TEN}.
+ *
+ * <p>This decision is orthogonal to the storage layout (e.g. LSM-tree); it
is keyed on the
+ * effective write version (i.e. {@code
HoodieWriteConfig#getWriteVersion()}) and base file format
+ * rather than the persisted table version, consistent with how the inline
log block layout is
+ * selected, so that the on-disk format follows what the writer is targeting
during
+ * upgrade/downgrade windows.
+ *
+ * @param writeConfig the writer configuration.
+ */
+ public static boolean shouldWriteNativeLogFormat(HoodieWriteConfig
writeConfig) {
Review Comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
