[
https://issues.apache.org/jira/browse/HUDI-8282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17889478#comment-17889478
]
Y Ethan Guo commented on HUDI-8282:
-----------------------------------
The log file marker on master / Hudi 1.0 is implemented through the
`LogFileCreationCallback` added to the `HoodieLogFormatWriter`. In
`HoodieWriteHandle`, the log writer is created with the log creation rollback.
{code:java}
HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.constructAbsolutePath(hoodieTable.getMetaClient().getBasePath(),
partitionPath))
.withFileId(fileId)
.withDeltaCommit(deltaCommitTime)
.withFileSize(0L)
.withSizeThreshold(config.getLogFileMaxSize())
.withStorage(storage)
.withRolloverLogWriteToken(writeToken)
.withLogWriteToken(writeToken)
.withFileCreationCallback(getLogCreationCallback())
.withSuffix(fileSuffix)
.withFileExtension(HoodieLogFile.DELTA_EXTENSION).build(); {code}
There are two other places that do not use the "getLogCreationCallback()" when
creating the log file writer:
{code:java}
HoodieBackedTableMetadataWriter
HoodieLogFormat.Writer writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.constructAbsolutePath(metadataWriteConfig.getBasePath(),
partitionName))
.withFileId(fileGroupFileId)
.withDeltaCommit(instantTime)
.withLogVersion(HoodieLogFile.LOGFILE_BASE_VERSION)
.withFileSize(0L)
.withSizeThreshold(metadataWriteConfig.getLogFileMaxSize())
.withStorage(dataMetaClient.getStorage())
.withRolloverLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
.withLogWriteToken(HoodieLogFormat.DEFAULT_WRITE_TOKEN)
.withFileExtension(HoodieLogFile.DELTA_EXTENSION).build()
BaseRollbackHelper
writer = HoodieLogFormat.newWriterBuilder()
.onParentPath(FSUtils.constructAbsolutePath(metaClient.getBasePath(),
rollbackRequest.getPartitionPath()))
.withFileId(fileId)
.withDeltaCommit(instantToRollback.getTimestamp())
.withStorage(metaClient.getStorage())
.withFileExtension(HoodieLogFile.DELTA_EXTENSION).build();{code}
> Create per log file marker for 1.x
> -----------------------------------
>
> Key: HUDI-8282
> URL: https://issues.apache.org/jira/browse/HUDI-8282
> Project: Apache Hudi
> Issue Type: Improvement
> Components: writer-core
> Reporter: sivabalan narayanan
> Assignee: sivabalan narayanan
> Priority: Blocker
> Fix For: 1.0.0
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> This ticket is meant for 1.x. We added per log file marker which was added to
> 0.x branch https://issues.apache.org/jira/browse/HUDI-1517
>
> Commit of interest:
> [https://github.com/apache/hudi/commit/c2c7e0538f8cf3031781ebdd776d1c03bfec3bb3]
>
>
> We might need to add a similar support to 1.x, hence the tracking ticket.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
