cshuo opened a new issue, #19287: URL: https://github.com/apache/hudi/issues/19287
## Problem Native log format can produce Lance data and delete logs (for example, `.log.lance` and `.deletes.lance`). Log compaction currently uses `HoodieAvroReaderContext` on executors, but Lance does not provide an Avro file reader or writer. A Spark reader context cannot currently be reconstructed reliably after the table is serialized to executors. As a result, native Lance log files cannot be read and rewritten by log compaction. ## Current mitigation `HoodieLogCompactionPlanGenerator` skips file slices containing native Lance logs so that an unexecutable log-compaction operation is not persisted to the timeline. ## Proposal Add native Lance log-compaction support by providing an executor-safe Spark Lance reader context and preserving Spark `InternalRow` records through the read and native Lance write paths. ## Acceptance criteria - Log compaction can read native `.log.lance` and `.deletes.lance` files on Spark executors. - Log compaction rewrites Lance native logs using Spark records and a Lance writer. - End-to-end coverage includes data and delete logs, including a scheduled-and-executed log compaction. - The scheduling guard can be removed once the path is supported. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
