nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720
thanks a lot for fixing this Dave. I would like to propose something here.
I am wondering why do we need to retrofit copying over delta streamer
checkpoint into hoodie.write.meta.key.prefixes. to me, this new requirement is
very simple and not really tied to `hoodie.write.meta.key.prefixes`.
Let me propose something and see how that looks like.
Introduce a new config called `hoodie.copy.over.deltastreamer.checkpoints`.
we can brainstorm on actual naming later.
When set to true,
within TransactionUtils::overrideWithLatestCommitMetadata
```
fetch value of "deltastreamer.checkpoint.key" from last committed
transaction and copy to cur inflight commit extra metadata.
```
This is very tight and not error prone. Users don't need to set two
different config as below which is not very intuitive as to why they need to do
this.
```
hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'
```
and optionally
```
deltastreamer.checkpoint.key =. ""
```
All users have to do is, for all of their spark writers, they need to set
`hoodie.copy.over.deltastreamer.checkpoints` to true.
welcome thoughts @n3nash @vinothchandar @davehagman
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
