[
https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14211496#comment-14211496
]
Vinoth Chandar commented on SQOOP-1168:
---------------------------------------
[~vybs] Good to know.. I am still learning the codebase, so don't have a
concrete design or anything yet.
My rough thoughts so far :
I am working on a sharded jdbc connector and my state for incremental import
would be a map of {shard_number => last_primary_key or last_modified_ts}. So,
at the code level, we might want to define classes (say
FromStateObject/ToStateObject), that encapsulate some state about the source &
destination over multiple runs. Sqoop can handle persisting this in the
repository and hand this object to the connector's Initializer (From/To), and
the connector write a new state objects after the job is a success.
Once this is in place, the GenericJdbcConnector can implement the two
incremental strategies as in sqoop1 and any more complicated connector is also
free to do its thing.
But super interested in this. we need this right out of the gate, for our use
cases.
> Sqoop2: Incremental Import
> --------------------------
>
> Key: SQOOP-1168
> URL: https://issues.apache.org/jira/browse/SQOOP-1168
> Project: Sqoop
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> Initial plan is to follow roughly the same design as Sqoop 1, except provide
> pluggability to start this through a REST API.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
