[
https://issues.apache.org/jira/browse/SQOOP-2579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934229#comment-14934229
]
Sqoop QA bot commented on SQOOP-2579:
-------------------------------------
Testing file
[SQOOP-2579.patch|https://issues.apache.org/jira/secure/attachment/12764101/SQOOP-2579.patch]
against branch sqoop2 took 0:56:46.233761.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} Clean was successful
{color:green}SUCCESS:{color} Patch applied correctly
{color:green}SUCCESS:{color} Patch add/modify test case
{color:green}SUCCESS:{color} License check passed
{color:green}SUCCESS:{color} Patch compiled
{color:green}SUCCESS:{color} All unit tests passed (executed 1428 tests)
{color:green}SUCCESS:{color} Test coverage did not decreased
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1735/artifact/patch-process/cobertura_report.txt])
{color:green}SUCCESS:{color} No new findbugs warnings
([report|https://builds.apache.org/job/PreCommit-SQOOP-Build/1735/artifact/patch-process/findbugs_report.txt])
{color:green}SUCCESS:{color} All integration tests passed (executed 89 tests)
Console output is available
[here|https://builds.apache.org/job/PreCommit-SQOOP-Build/1735/console].
This message is automatically generated.
> Sqoop2: Refactore RepositoryManager to not load structure from repository
> when loading jobs and links
> -----------------------------------------------------------------------------------------------------
>
> Key: SQOOP-2579
> URL: https://issues.apache.org/jira/browse/SQOOP-2579
> Project: Sqoop
> Issue Type: Bug
> Affects Versions: 1.99.6
> Reporter: Jarek Jarcec Cecho
> Assignee: Abraham Fine
> Attachments: SQOOP-2579.patch
>
>
> This one will take a bit longer to explain, so please _beer_ with me :)
> The current flow of configuration class is this one:
> * Connector developer defines class and marks inputs that are required from
> users with
> [{{@Input}}|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/model/Input.java]
> annotation. This annotation can have several parameters ({{sensitive}},
> {{size}}, ...).
> * As we can't use those classes everywhere (mainly we can't send them to
> clients over the wire nor serialize them to repository), we have
> [{{ConfigUtil}}|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/model/ConfigUtils.java]
> class that serializes them to [model
> objects|https://github.com/apache/sqoop/tree/sqoop2/common/src/main/java/org/apache/sqoop/model].
> * Now {{ConnectorManager}} takes the model objects and persist them in memory
> for the lifetime of Sqoop 2 server process (after restart we simply run the
> {{ConfigUtils}} again, so nothing is lost).
> * The {{RepositoryManager}} takes the model objects and persist them in
> database. This is desirable because this way we can detect if the structures
> actually changed (e.g. connector code has been updated) and we can call
> upgrade routines.
> On REST layer, we have following behavior:
> * For {{/connector/}} we always call
> [{{ConnectorManager}}|https://github.com/apache/sqoop/blob/sqoop2/server/src/main/java/org/apache/sqoop/handler/ConnectorRequestHandler.java]
> to get the in memory structures.
> * For {{/job}} and {{/links}} - e.g. to get actual values for those model
> objects - we are using
> [{{RepositoryManager}}|https://github.com/apache/sqoop/blob/sqoop2/server/src/main/java/org/apache/sqoop/handler/JobRequestHandler.java]
> instead. As the {{RepositoryManager}} is currently not using
> {{ConnectorManager}} it uses the model objects that are serialized in the
> database instead.
> This behavior is a bit limiting because now if we want to improve the model
> classes and add new information
> ([Validations|https://issues.apache.org/jira/browse/SQOOP-1442], [Sensitive
> keys for maps|https://issues.apache.org/jira/browse/SQOOP-2549]) we
> absolutely have to serialize that information in database. And that means
> that we have to do database upgrade, that means that if this information is
> changed we have to do data upgrade, ... Even though that this is not required
> as those kind of things can be kept in memory as we don't need to do any
> action when they change.
> Hence I would like to propose changing current behavior and making
> {{RepositoryManager}} use {{ConnectroManager}} to read jobs and links
> objects. This way we can have more in memory state and what state is
> persisted in database.
> Please note that I'm not suggesting to drop the model classes from repository
> - we still need them to detect if metadata (the {{@Input}} annotation) has
> changed and run upgrade if required. I'm only suggesting to not used this
> serialized metadata if we are reading job and links objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
