[
https://issues.apache.org/jira/browse/SQOOP-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Veena Basavaraj updated SQOOP-1799:
-----------------------------------
Description:
One suggestion would be have a connector's FROM/ TO initializer to expose if it
even supports incremental. So this can be used to immediately validate the job
creation.
{code}
sqoop > create incremental-job -f 1 -t 2
{code}
HDFS FROM supporting incrementation read ? Does this even apply. But surely the
TO side should support the delta/ incremental write.
Both the from connector and to connector has to support this feature before we
proceed. The default will be false. The Initializer API will be updated to
support this.
{code}
import java.util.LinkedList;
import java.util.List;
import org.apache.sqoop.schema.NullSchema;
import org.apache.sqoop.schema.Schema;
/**
* This allows connector to define initialization work for execution,
* for example, context configuration.
*/
public abstract class Initializer<LinkConfiguration, JobConfiguration> {
/**
* Initialize new submission based on given configuration properties. Any
* needed temporary values might be saved to context object and they will be
* promoted to all other part of the workflow automatically.
*
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
*/
public abstract void initialize(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration);
/**
* Return list of all jars that this particular connector needs to operate on
* following job. This method will be called after running initialize method.
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
* @return
*/
public List<String> getJars(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration) {
return new LinkedList<String>();
}
/**
* Return schema associated with the connector for FROM and TO
* By default we assume a null schema. Override the method if there a custom
schema to provide either for FROM or TO
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
* @return
*/
public Schema getSchema(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration) {
return NullSchema.getInstance();
}
{code}
was:
One suggestion would be have a connector's FROM/ TO initializer to expose if it
even supports incremental. So this can be used to immediately validate the job
creation.
{code}
sqoop > create incremental-job -f 1 -t 2
{code}
Both the from connector and to connector has to support this feature before we
proceed. The default will be false. The Initializer API will be updated to
support this.
{code}
import java.util.LinkedList;
import java.util.List;
import org.apache.sqoop.schema.NullSchema;
import org.apache.sqoop.schema.Schema;
/**
* This allows connector to define initialization work for execution,
* for example, context configuration.
*/
public abstract class Initializer<LinkConfiguration, JobConfiguration> {
/**
* Initialize new submission based on given configuration properties. Any
* needed temporary values might be saved to context object and they will be
* promoted to all other part of the workflow automatically.
*
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
*/
public abstract void initialize(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration);
/**
* Return list of all jars that this particular connector needs to operate on
* following job. This method will be called after running initialize method.
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
* @return
*/
public List<String> getJars(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration) {
return new LinkedList<String>();
}
/**
* Return schema associated with the connector for FROM and TO
* By default we assume a null schema. Override the method if there a custom
schema to provide either for FROM or TO
* @param context Initializer context object
* @param linkConfiguration link configuration object
* @param jobConfiguration job configuration object for the FROM and TO
* In case of the FROM initializer this will represent the FROM job
configuration
* In case of the TO initializer this will represent the TO job
configuration
* @return
*/
public Schema getSchema(InitializerContext context, LinkConfiguration
linkConfiguration,
JobConfiguration jobConfiguration) {
return NullSchema.getInstance();
}
{code}
> Ability for connector to indicate if its FROM and TO support incremental
> reading/ writing
> -----------------------------------------------------------------------------------------
>
> Key: SQOOP-1799
> URL: https://issues.apache.org/jira/browse/SQOOP-1799
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 1.99.5
>
>
> One suggestion would be have a connector's FROM/ TO initializer to expose if
> it even supports incremental. So this can be used to immediately validate the
> job creation.
> {code}
> sqoop > create incremental-job -f 1 -t 2
> {code}
> HDFS FROM supporting incrementation read ? Does this even apply. But surely
> the TO side should support the delta/ incremental write.
> Both the from connector and to connector has to support this feature before
> we proceed. The default will be false. The Initializer API will be updated to
> support this.
> {code}
> import java.util.LinkedList;
> import java.util.List;
> import org.apache.sqoop.schema.NullSchema;
> import org.apache.sqoop.schema.Schema;
> /**
> * This allows connector to define initialization work for execution,
> * for example, context configuration.
> */
> public abstract class Initializer<LinkConfiguration, JobConfiguration> {
> /**
> * Initialize new submission based on given configuration properties. Any
> * needed temporary values might be saved to context object and they will be
> * promoted to all other part of the workflow automatically.
> *
> * @param context Initializer context object
> * @param linkConfiguration link configuration object
> * @param jobConfiguration job configuration object for the FROM and TO
> * In case of the FROM initializer this will represent the FROM job
> configuration
> * In case of the TO initializer this will represent the TO job
> configuration
> */
> public abstract void initialize(InitializerContext context,
> LinkConfiguration linkConfiguration,
> JobConfiguration jobConfiguration);
> /**
> * Return list of all jars that this particular connector needs to operate
> on
> * following job. This method will be called after running initialize
> method.
> * @param context Initializer context object
> * @param linkConfiguration link configuration object
> * @param jobConfiguration job configuration object for the FROM and TO
> * In case of the FROM initializer this will represent the FROM job
> configuration
> * In case of the TO initializer this will represent the TO job
> configuration
> * @return
> */
> public List<String> getJars(InitializerContext context, LinkConfiguration
> linkConfiguration,
> JobConfiguration jobConfiguration) {
> return new LinkedList<String>();
> }
> /**
> * Return schema associated with the connector for FROM and TO
> * By default we assume a null schema. Override the method if there a
> custom schema to provide either for FROM or TO
> * @param context Initializer context object
> * @param linkConfiguration link configuration object
> * @param jobConfiguration job configuration object for the FROM and TO
> * In case of the FROM initializer this will represent the FROM job
> configuration
> * In case of the TO initializer this will represent the TO job
> configuration
> * @return
> */
> public Schema getSchema(InitializerContext context, LinkConfiguration
> linkConfiguration,
> JobConfiguration jobConfiguration) {
> return NullSchema.getInstance();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
