vinothchandar commented on PR #5436: URL: https://github.com/apache/hudi/pull/5436#issuecomment-1143251715
That said - I am wondering if we can avoid 1. Adding the new metafield `_hoodie_operation ` by default or thinking about all schema evolution pains. 2. Adding a new field or something to hold the before image within the base parquet file. To that extent, how about we define a new CDC Block, that is always populated in the log format even for COW table. i.e if user invokes MergeHandle on COW table, we can do one of the following. 1. [Streaming friendly] Write out the new base file as usual, but additionally add a new log file, that contains a single CDC block with the `_hoodie_operation` values for the base file. This will be a pretty small file to read, overhead-wise. I just moved it from being within a meta field to being logged separately. 2. [Batch friendly] Write out the new base file as usual, but diff this against the last base file and add a new log file. This time the CDC block also contains the entire before image (by doing the diffing at write time and materializing results), in addition to the `_hoodie_operation` thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
