[
https://issues.apache.org/jira/browse/SQOOP-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180249#comment-14180249
]
Gwen Shapira commented on SQOOP-1603:
-------------------------------------
Also, disagree that Destroy should only be used for cleanup. It is just the
final step of the ETL process.
Why not use it for whatever steps the TO connector wants to do once the data
load is finished?
I can think of a lot of valid applications in RDBMS: Partition swapping,
creating indexes, rebuilding indexes, collecting statistics on tables,
reporting statistics about the load to Kafka, etc, etc.
> Sqoop2: Explicit support for Merge in the Sqoop Job lifecyle
> --------------------------------------------------------------
>
> Key: SQOOP-1603
> URL: https://issues.apache.org/jira/browse/SQOOP-1603
> Project: Sqoop
> Issue Type: Bug
> Reporter: Veena Basavaraj
> Assignee: Qian Xu
>
> This ticket was created while reviewing the Kite Connector use case where the
> destroyer does the actual temp data set merge
> https://reviews.apache.org/r/26963/diff/# [~stanleyxu2005]
> {code}
> public void destroy(DestroyerContext context, LinkConfiguration link,
> ToJobConfiguration job) {
> LOG.info("Running Kite connector destroyer");
> // Every loader instance creates a temporary dataset. If the MR job is
> // successful, all temporary dataset should be merged as one dataset,
> // otherwise they should be deleted all.
> String[] uris = KiteDatasetExecutor.listTemporaryDatasetUris(
> job.toDataset.uri);
> if (context.isSuccess()) {
> KiteDatasetExecutor executor = new
> KiteDatasetExecutor(job.toDataset.uri,
> context.getSchema(), link.link.fileFormat);
> for (String uri : uris) {
> executor.mergeDataset(uri);
> LOG.info(String.format("Temporary dataset %s merged", uri));
> }
> } else {
> for (String uri : uris) {
> KiteDatasetExecutor.deleteDataset(uri);
> LOG.info(String.format("Temporary dataset %s deleted", uri));
> }
> }
> }
> {code}
> Wondering if such things should be its own phase rather than in destroyers.
> The responsibility of destroyer is more to clean up/ closing/ anything thats
> pretty much destroying, should such operations that modify records its own
> step.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
