sivabalan narayanan created HUDI-2719:
-----------------------------------------
Summary: Add back support for copying over extra metadata from
previous commit
Key: HUDI-2719
URL: https://issues.apache.org/jira/browse/HUDI-2719
Project: Apache Hudi
Issue Type: Improvement
Reporter: sivabalan narayanan
Add back support for copying over extra metadata from previous commit .
We had a config named 'hoodie.write.meta.key.prefixes'. when this config is
added, hudi copies over any extra metadata from previous commit with matching
key prefixes as per `hoodie.write.meta.key.prefixes`. for eg: if
`hoodie.write.meta.key.prefixes` set to `abc` and if there are other config
options as "abc.key1" and "abc.key2", these two will be copied over from
previous commit metadata and get overridden with current writer's commit
metadata.
Our initial understanding is that this was added to support copying over delta
streamer checkpoint for multi -writer scenairo. But since the config is not
very straight forward to use, we simplified and introduced a new config as per
[https://github.com/apache/hudi/pull/3820.]
Added the old one "hoodie.write.meta.key.prefixes" as an alternative. Since
multi-writer in itself is still evolving, we felt simplifying the config for
deltastreamer checkpoint makes sense.
But if someone out there was using the older config to copy some other extra
metadata for debugging or some other purposes, we need to add the support back.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
