[
https://issues.apache.org/jira/browse/SQOOP-1804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258449#comment-14258449
]
Veena Basavaraj edited comment on SQOOP-1804 at 12/24/14 6:56 PM:
------------------------------------------------------------------
Having a key declared upfront does not enforce any consistency since we cannot
validate the code in the connector actually does.
This is true with the configs as well, I can declare a key "foobar" and not use
it in the code, so I am not sure what is the consistency part been referred to,
but it is more of a I want to know upfront looking at the config class what are
the state variables that a connector exposes, then it surely adds value to the
user looking at the connector code and the end user using the connector.
My original proposal was to treat state as config and annotate it as INTERNAL
MInputType ( see #1 in the design wiki) as well, which meant that you get
everything you have configs today with state as well, if declaring in advance
is a requirement. In that case there is absolutely no wok for this ticket,
except adding a new type that the value will be annotated with
So if we want the "state" to be treated like "configs" to allow for upgrade
etc, then we should do it, it is absolutely no effort.
we can provide a STATE/OUPUT annotation and store such things in the existing
SQ_CONFIG table or add a new one SQ_STATE that mirrors SQ_INPUT if want to be
semantically separating, input from output.
{code}
@ConfigClass(validators = {
@Validator(DeltaFetchConfig1.DeltaFetchConfig1Validator.class)})
public class DeltaFetchConfig1 {
@Input(size = 255) public String column;
// validate supported oprators, can provide a default value etc...
@Input public String operator;
@Input public String value;
@Input public String value;
@State public String foobar or @Output public String foobar
}
{code}
was (Author: vybs):
Having a key declared upfront does not enforce any consistency since we cannot
validate the code in the connector actually does.
This is true with the configs as well, I can declare a key "foobar" and not use
it in the code, so I am not sure what is the consistency part been referred to,
but it is more of a I want to know upfront looking at the config class what are
the state variables that a connector exposes, then it surely adds value to the
user looking at the connector code and the end user using the connector.
My original proposal was to treat state as config and annotate it as INTERNAL
MInputType ( see #1 in the design wiki) as well, which meant that you get
everything you have configs today with state as well, if declaring in advance
is a requirement. In that case there is absolutely no wok for this ticket,
except adding a new type that the value will be annotated with
So if we want the "state" to be treated like "configs" to allow for upgrade
etc, then we should do it, it is absolutely no effort.
we can provide a STATE/OUPUT annotation and store such things in the existing
SQ_CONFIG table or add a new one SQ_STATE that mirrors SQ_INPUT if want to be
semantically separating, input from output.
{code}
@ConfigClass(validators = {
@Validator(DeltaFetchConfig1.DeltaFetchConfig1Validator.class)})
public class DeltaFetchConfig1 {
@Input(size = 255) public String column;
// validate supported oprators, can provide a default value etc...
@Input public String operator;
@Input public String value;
@Input public String value;
@State public String foobar or @Ouput public String foobar
}
{code}
> Repository Structure + API: Storing/Retrieving the From/To state of the
> incremental read/ write
> -----------------------------------------------------------------------------------------------
>
> Key: SQOOP-1804
> URL: https://issues.apache.org/jira/browse/SQOOP-1804
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> Details of this proposal are in the wiki.
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+And+Merge+Design#DeltaFetchAndMergeDesign-Wheretostoretheoutputinsqoop?
> Update: The above highlights the pros and cons of each approach.
> #4 is chosen, since it is less intrusive, more clean and allows U/Edit per
> value in the output easily.
> Will use this ticket for more detailed discussion on storage options for the
> output from connectors
> 1.
> {code}
> // will have FK to submission
> public static final String QUERY_CREATE_TABLE_SQ_JOB_OUTPUT_SUBMISSION =
> "CREATE TABLE " + TABLE_SQ_JOB_OUTPUT + " ("
> + COLUMN_SQ_JOB_OUT_ID + " BIGINT GENERATED ALWAYS AS IDENTITY (START
> WITH 1, INCREMENT BY 1), "
> + COLUMN_SQ_JOB_OUT_KEY + " VARCHAR(32), "
> + COLUMN_SQ_JOB_OUT_VALUE + " LONG VARCHAR,"
> + COLUMN_SQ_JOB_OUT_TYPE + " VARCHAR(32),"
> + COLUMN_SQD_ID + " VARCHAR(32)," // FK to the direction table, since
> this allows to distinguish output from FROM/ TO part of the job
> + COLUMN_SQRS_SUBMISSION + " BIGINT, "
> + "CONSTRAINT " + CONSTRAINT_SQRS_SQS + " "
> + "FOREIGN KEY (" + COLUMN_SQRS_SUBMISSION + ") "
> + "REFERENCES " + TABLE_SQ_SUBMISSION + "(" + COLUMN_SQS_ID + ") ON
> DELETE CASCADE "
> {code}
> 2.
> At the code level, we will define MOutputType, one of the types can be BLOB
> as well, if a connector decides to store the value as a BLOB
> {code}
> class JobOutput {
> String key;
> Object value;
> MOutputType type;
> }
> {code}
> 3.
> At the repository API, add a new API to get job output for a particular
> submission Id and allow updates on values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
