[
https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251135#comment-14251135
]
Veena Basavaraj edited comment on SQOOP-1168 at 12/18/14 4:05 AM:
------------------------------------------------------------------
[~gwenshap]
>>>Extra clarification about use of COUNTER:
I prefer doing the right thing, than the easy thing. If submission is the
place we want to store it seems apt, BTW the right thing is as not that hard
either, adding a table and extending the current table to support more types is
not that hard, I have done many changes in the repo so far so I speak from that
experience.:). the wiki explains the details of the change
>>in this case, what's the plan on validating that the inputs for "from" and
>>"to" connectors are consistent?
I am not sure what consistent means? The connectors have a api in initializer
that says if they support delta fetch / merge.
[~jarcec] and I were discussing this offline, so will repeat the same here
There are few things that can happen
create job -f 1 -t 2
1. f and t both dont support, all good, since there wont be any such configs
displayed
2. f supports delta and t supports delta, all good
For the following two
3. f supports delta and t does not,
4. f does not support, but t does.
We can do upfront check and do not display the delta configs at all, not that
hard to do ( that I prefer )
or we can be more flexible and say that if the t supports delta, then lets its
ok or f supports delta and t does not, we cannot guarantee how it will written
Very similar to schema matching, where the rules have been defined based on the
current assumptions that LocationMatcher takes precedence over Name matching.
Update: One more thing about consistency...
AFAI see it, consistency is not something sqoop needs to be worried about, at
the end of FROM, it gives a set of records and TO will take it try to write it
in the ways it supports, else it will default to the case it commonly will
use.
If there is concrete use case, kindly provide it.
was (Author: vybs):
[~gwenshap]
>>>Extra clarification about use of COUNTER:
I prefer doing the right thing, than the easy thing. If submission is the
place we want to store it seems apt, BTW the right thing is as not that hard
either, adding a table and extending the current table to support more types is
not that hard, I have done many changes in the repo so far so I speak from that
experience.:). the wiki explains the details of the change
>>n this case, what's the plan on validating that the inputs for "from" and
>>"to" connectors are consistent?
I am not sure what consistent means? The connectors have a api in initializer
that says if they support delta fetch / merge.
[~jarcec] and I were discussing this offline, so will repeat the same here
There are few things that can happen
create job -f 1 -t 2
f and t both dont support, all good, since there wont be any such configs
displayed
f supports delta and t supports delta, all good
For the following two
f supports delta and t does not,
f does not support, but t does.
we can do upfront check and do not display the delta configs at all, not that
hard to do ( that I prefer )
or we can be more flexible and say if the t supports delta, then lets its ok.
very similar to schema matching, where the rules have been defined based on the
current assumptions that LocationMatcher takes precedence over Name matching.
> Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )
> -----------------------------------------------------------------
>
> Key: SQOOP-1168
> URL: https://issues.apache.org/jira/browse/SQOOP-1168
> Project: Sqoop
> Issue Type: Bug
> Reporter: Hari Shreedharan
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> The formal design wiki is here
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+and+Merge+Design
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
