[
https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251793#comment-14251793
]
Veena Basavaraj edited comment on SQOOP-1168 at 12/18/14 3:30 PM:
------------------------------------------------------------------
[~gwenshap] I might just implement it and send it, since I did highlight that
the way to do this is to put a value in the context objects. One way is to use
a standardized KEY (CONNECTOR_OUPUT_ so we know that if that key exists then
it has the output that I want to store on the repo, as simple as that, there is
nothing more to it.
My goal is not to introduce anything specific to delta or incremental, later on
this should extend to write any other info from connector to repository via
sqoop. There is nothing special about last value or rows read or any other
foobar
hope this makes sense. This was explained in the design wiki already ...
{quote}
Storing state (2 options)
See #2 in open question on how data from connector will flow into sqoop and
then to repository.
We can store the values from the Extractor context and Loader Context, that are
passed to sqoop in 2 places
SQ_JOB_INPUT ( with a new MInputType, INTERNAL, or other relevant name to say
that this is not what connectors exposed, but resulted from JOB execution for
FROM and TO), the real advantage is that we have separation per CONFIG per
DIRECTION.
We can dump everything in SQ_COUNTER_SUBMISSION
related to submission, but what we may want to give back may be not always
counter or long value, it can be a boolean saying if it was a delta fetch or
merge , it wont have the distinction per direction. If we are to go this route,
the SQ_COUNTER_SUBMISSION needs toe generalize to support any form of job
output values to be stored in it and exposed via submission API. A simiple way
to think about it is to mirror what we have in MInputType for MOutputType
{quote}
was (Author: vybs):
[~gwenshap] I might just implement it and send it, since I did highlight that
the way to do this is to put a value in the context objects. One way is to use
a standardized KEY (CONNECTOR_OUPUT_ so we know that if that key exists then
it has the output that I want to store on the repo, as simple as that, there is
nothing more to it.
My goal is not to introduce anything specific to delta or incremental, later on
this should extend to write any other info from connector to repository via
sqoop. There is nothing special about last value or rows read or any other
foobar
hope this makes sense. This was explained in the design wiki already
> Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )
> -----------------------------------------------------------------
>
> Key: SQOOP-1168
> URL: https://issues.apache.org/jira/browse/SQOOP-1168
> Project: Sqoop
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> The formal design wiki is here
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+and+Merge+Design
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
