danny0405 commented on code in PR #6256:
URL: https://github.com/apache/hudi/pull/6256#discussion_r960403535
##########
rfc/rfc-51/rfc-51.md:
##########
@@ -148,20 +152,46 @@ hudi_cdc_table/
Under a partition directory, the `.log` file with `CDCBlock` above will keep
the changing data we have to materialize.
-There is an option to control what data is written to `CDCBlock`, that is
`hoodie.table.cdc.supplemental.logging`. See the description of this config
above.
+#### Persisting CDC in MOR: Write-on-indexing vs Write-on-compaction
+
+2 design choices on when to persist CDC in MOR tables:
+
+Write-on-indexing allows CDC info to be persisted at the earliest, however, in
case of Flink writer or Bucket
+indexing, `op` (I/U/D) data is not available at indexing.
+
+Write-on-compaction can always persist CDC info and achieve standardization of
implementation logic across engines,
+however, some delays are added to the CDC query results. Based on the business
requirements, Log Compaction (RFC-48) or
+scheduling more frequent compaction can be used to minimize the latency.
-Spark DataSource example:
+The semantics we propose to establish are: when base files are written, the
corresponding CDC data is also persisted.
+
+- For Spark
+ - inserts are written to base files: the CDC data `op=I` will be persisted
+ - updates/deletes that written to log files are compacted into base files:
the CDC data `op=U|D` will be persisted
+- For Flink
+ - inserts/updates/deletes that written to log files are compacted into base
files: the CDC data `op=I|U|D` will be
+ persisted
+
Review Comment:
> What do you suggest on when we write out CDC log blocks when we use bucket
index
Does the index type (bloom filter/bucket) affect the CDC log blocks here ?
Personally i think generating the log blocks during write on the fly is not
very feasible because looking up the history records index would cost much
which reduce the write throughput.
We can generate the cdc on the fly for the mor table reader, and the CDC log
blocks generated by compaction is a optimization view like the cow table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
