[
https://issues.apache.org/jira/browse/HUDI-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368640#comment-17368640
]
Sagar Sumit commented on HUDI-1860:
-----------------------------------
[~samrat007] Thanks for your interest! Feel free to assign this sub-task to
yourself.
For context, we recently introduced
[JdbcSource|https://github.com/apache/hudi/blob/master/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/JdbcSource.java]
for DeltaStreamer. It can run in two modes, namely, full fetch and incremental
mode. Full fetch is like `select * from table` kind of query, while the
incremental fetch keeps pulling incrementally from a checkpoint continuously.
As mentioned in RFC-14, it would be nice to have the full fetch use the
INSERT_OVERWRITE operation, that way, it can also handle schema changes.
However, we need to extend the insert_overwrite support to DeltaStreamer and
that is the goal of this sub-task. Take a look at DeltaSync::writeToSink method.
> Add INSERT_OVERWRITE support to DeltaStreamer
> ---------------------------------------------
>
> Key: HUDI-1860
> URL: https://issues.apache.org/jira/browse/HUDI-1860
> Project: Apache Hudi
> Issue Type: Sub-task
> Reporter: Sagar Sumit
> Assignee: Sagar Sumit
> Priority: Major
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> As discussed in [this
> RFC|https://cwiki.apache.org/confluence/display/HUDI/RFC+-+14+%3A+JDBC+incremental+puller],
> having full fetch mode use the inser_overwrite to write to sync would be
> better as it can handle schema changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
