git commit: SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector
Repository: sqoop Updated Branches: refs/heads/SQOOP-1367 9f8f59514 - ca16703e4 SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector (Abraham Elmahrek via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/ca16703e Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/ca16703e Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/ca16703e Branch: refs/heads/SQOOP-1367 Commit: ca16703e418b49c71eb8371833e6a569c61014cf Parents: 9f8f595 Author: Jarek Jarcec Cecho jar...@apache.org Authored: Thu Oct 9 08:16:52 2014 -0700 Committer: Jarek Jarcec Cecho jar...@apache.org Committed: Thu Oct 9 08:16:52 2014 -0700 -- docs/src/site/sphinx/CommandLineClient.rst | 6 +- docs/src/site/sphinx/ConnectorDevelopment.rst | 154 ++- docs/src/site/sphinx/RESTAPI.rst| 54 +-- .../sqoop/connector/spi/SqoopConnector.java | 2 +- 4 files changed, 95 insertions(+), 121 deletions(-) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/ca16703e/docs/src/site/sphinx/CommandLineClient.rst -- diff --git a/docs/src/site/sphinx/CommandLineClient.rst b/docs/src/site/sphinx/CommandLineClient.rst index fb7f780..7cc6449 100644 --- a/docs/src/site/sphinx/CommandLineClient.rst +++ b/docs/src/site/sphinx/CommandLineClient.rst @@ -329,14 +329,14 @@ Create new job object. ++--+ | Argument | Description | ++==+ -| ``-x``, ``--xid x`` | Create new job object for connection with id ``x`` | +| ``-f``, ``--from x`` | Create new job object with a FROM connection with id ``x`` | ++--+ -| ``-t``, ``--type t`` | Create new job object with type ``t`` (``import``, ``export``) | +| ``-t``, ``--to t`` | Create new job object with a TO connection with id ``x`` | ++--+ Example: :: - create job --xid 1 + create job --from 1 --to 2 Update Command -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/ca16703e/docs/src/site/sphinx/ConnectorDevelopment.rst -- diff --git a/docs/src/site/sphinx/ConnectorDevelopment.rst b/docs/src/site/sphinx/ConnectorDevelopment.rst index ae4f721..d700e4c 100644 --- a/docs/src/site/sphinx/ConnectorDevelopment.rst +++ b/docs/src/site/sphinx/ConnectorDevelopment.rst @@ -26,17 +26,15 @@ using the code of built-in connector ( ``GenericJdbcConnector`` ) as example. What is Connector? ++ -Connector provides interaction with external databases. -Connector reads data from databases for import, -and write data to databases for export. -Interaction with Hadoop is taken cared by common modules of Sqoop 2 framework. +The connector provides the facilities to interact with external data sources. +The connector can read from, or write to, a data source. When do we add a new connector? === You add a new connector when you need to extract data from a new data source, or load data to a new target. In addition to the connector API, Sqoop 2 also has an engine interface. -At the moment the only engine is MapReduce,but we may support additional engines in the future. +At the moment the only engine is MapReduce, but we may support additional engines in the future. Since many parallel execution engines are capable of reading/writing data there may be a question of whether support for specific data stores should be done through a new connector or new engine. @@ -51,57 +49,73 @@ Connector Implementation The ``SqoopConnector`` class defines functionality which must be provided by Connectors. -Each Connector must extends ``SqoopConnector`` and overrides methods shown below. +Each Connector must extend ``SqoopConnector`` and override the methods shown below. :: public abstract String getVersion(); public abstract ResourceBundle getBundle(Locale locale); - public abstract Class getConnectionConfigurationClass(); - public abstract Class getJobConfigurationClass(MJob.Type jobType); - public abstract Importer getImporter(); - public abstract Exporter getExporter(); + public abstract Class getLinkConfigurationClass(); + public abstract Class getJobConfigurationClass(Direction direction); + public abstract From getFrom(); + public abstract To getTo();
git commit: SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
Repository: sqoop
Updated Branches:
refs/heads/SQOOP-1367 ca16703e4 - 3391e3950
SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/3391e395
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/3391e395
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/3391e395
Branch: refs/heads/SQOOP-1367
Commit: 3391e39501688ff6f2dde8e43b965be4dcff9440
Parents: ca16703
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Thu Oct 9 08:36:35 2014 -0700
Committer: Jarek Jarcec Cecho jar...@apache.org
Committed: Thu Oct 9 08:36:35 2014 -0700
--
.../sqoop/common/SupportedDirections.java | 79 +++-
.../sqoop/common/TestSupportedDirections.java | 19 +
.../sqoop/repository/derby/DerbyRepoError.java | 8 +
.../derby/DerbyRepositoryHandler.java | 398 ---
.../repository/derby/DerbySchemaConstants.java | 53 +++
.../repository/derby/DerbySchemaQuery.java | 164 +++-
.../sqoop/repository/derby/DerbyTestCase.java | 131 --
.../repository/derby/TestConnectorHandling.java | 62 ++-
.../sqoop/repository/derby/TestJobHandling.java | 4 -
.../sqoop/shell/ShowConnectorFunction.java | 30 +-
.../sqoop/connector/spi/SqoopConnector.java | 2 +-
11 files changed, 808 insertions(+), 142 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3391e395/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
index 25ba276..c527117 100644
--- a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
+++ b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
@@ -20,7 +20,9 @@ package org.apache.sqoop.common;
/**
* Represents which Directions are supported.
*/
-public class SupportedDirections {
+public class SupportedDirections implements ComparableSupportedDirections {
+ private static final char SUPPORTED_DIRECTIONS_SEPARATOR = '/';
+
private boolean from;
private boolean to;
@@ -38,4 +40,79 @@ public class SupportedDirections {
return direction == Direction.FROM from
|| direction == Direction.TO to;
}
+
+ /**
+ * @return String FROM, TO, FROM/TO, .
+ */
+ public String toString() {
+StringBuffer buffer = new StringBuffer();
+
+if (isDirectionSupported(Direction.FROM)) {
+ buffer.append(Direction.FROM);
+
+ if (isDirectionSupported(Direction.TO)) {
+buffer.append(SUPPORTED_DIRECTIONS_SEPARATOR);
+buffer.append(Direction.TO);
+ }
+} else if (isDirectionSupported(Direction.TO)) {
+ buffer.append(Direction.TO);
+}
+
+return buffer.toString();
+ }
+
+ public static SupportedDirections fromString(String supportedDirections) {
+boolean from = false, to = false;
+
+if (supportedDirections != null !supportedDirections.equals()) {
+ for (String direction : supportedDirections.split(/)) {
+switch (Direction.valueOf(direction)) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+ }
+}
+
+return new SupportedDirections(from, to);
+ }
+
+ public static SupportedDirections fromDirection(Direction direction) {
+boolean from = false, to = false;
+switch (direction) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+return new SupportedDirections(from, to);
+ }
+
+ @Override
+ public int compareTo(SupportedDirections o) {
+int hash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ hash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ hash |= 2;
+}
+
+int oHash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ oHash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ oHash |= 2;
+}
+
+return hash - oHash;
+ }
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3391e395/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
--
diff --git
a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
index 4fbaf82..4f0cdd6 100644
--- a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
+++ b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
@@ -52,4 +52,23 @@
svn commit: r1630590 - in /sqoop/site/trunk/content/resources/doap: ./ doap_Sqoop.rdf
Author: jarcec Date: Thu Oct 9 20:34:10 2014 New Revision: 1630590 URL: http://svn.apache.org/r1630590 Log: Adding DOAP file Added: sqoop/site/trunk/content/resources/doap/ sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf Added: sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf?rev=1630590view=auto == --- sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf (added) +++ sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf Thu Oct 9 20:34:10 2014 @@ -0,0 +1,56 @@ +?xml version=1.0? +?xml-stylesheet type=text/xsl? +rdf:RDF xml:lang=en + xmlns=http://usefulinc.com/ns/doap#; + xmlns:rdf=http://www.w3.org/1999/02/22-rdf-syntax-ns#; + xmlns:asfext=http://projects.apache.org/ns/asfext#; + xmlns:foaf=http://xmlns.com/foaf/0.1/; +!-- +Licensed to the Apache Software Foundation (ASF) under one or more +contributor license agreements. See the NOTICE file distributed with +this work for additional information regarding copyright ownership. +The ASF licenses this file to You under the Apache License, Version 2.0 +(the License); you may not use this file except in compliance with +the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an AS IS BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +-- + Project rdf:about=http://sqoop.apache.org; +created2014-09-21/created +license rdf:resource=http://spdx.org/licenses/Apache-2.0; / +nameApache Sqoop/name +homepage rdf:resource=http://sqoop.apache.org; / +asfext:pmc rdf:resource=http://sqoop.apache.org; / +shortdescApache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases./shortdesc +bug-database rdf:resource=http://issues.apache.org/jira/browse/SQOOP; / +mailing-list rdf:resource=http://sqoop.apache.org/mail-lists.html; / +download-page rdf:resource=http://sqoop.apache.org/; / +programming-languageJava/programming-language +category rdf:resource=http://projects.apache.org/category/big-data; / +release + Version +nameApache Sqoop 1.4.5/name +created2014-08-20/created +revision1.4.5/revision + /Version +/release +repository + GitRepository +location rdf:resource=https://git-wip-us.apache.org/repos/asf/sqoop.git/ +browse rdf:resource=https://git-wip-us.apache.org/repos/asf?p=sqoop.git/ + /GitRepository +/repository +maintainer + foaf:Person +foaf:nameApache Sqoop/foaf:name + foaf:mbox rdf:resource=mailto:d...@sqoop.apache.org/ + /foaf:Person +/maintainer + /Project +/rdf:RDF
Git Push Summary
Repository: sqoop Updated Branches: refs/heads/doap [deleted] bda201252
svn commit: r925198 - /websites/staging/sqoop/trunk/content/
Author: buildbot Date: Thu Oct 9 20:39:49 2014 New Revision: 925198 Log: Staging update by buildbot for sqoop Modified: websites/staging/sqoop/trunk/content/ (props changed) Propchange: websites/staging/sqoop/trunk/content/ -- --- cms:source-revision (original) +++ cms:source-revision Thu Oct 9 20:39:49 2014 @@ -1 +1 @@ -1630591 +1630592
svn commit: r1630592 - /sqoop/site/trunk/pom.xml
Author: jarcec Date: Thu Oct 9 20:39:35 2014 New Revision: 1630592 URL: http://svn.apache.org/r1630592 Log: Another whitespace branch to push staging build Modified: sqoop/site/trunk/pom.xml Modified: sqoop/site/trunk/pom.xml URL: http://svn.apache.org/viewvc/sqoop/site/trunk/pom.xml?rev=1630592r1=1630591r2=1630592view=diff == --- sqoop/site/trunk/pom.xml (original) +++ sqoop/site/trunk/pom.xml Thu Oct 9 20:39:35 2014 @@ -15,7 +15,7 @@ See the License for the specific language governing permissions and limitations under the License. --- +-- project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd; modelVersion4.0.0/modelVersion
svn commit: r925199 - /websites/production/sqoop/content/
Author: jarcec Date: Thu Oct 9 20:40:04 2014 New Revision: 925199 Log: d Added: websites/production/sqoop/content/ - copied from r925198, websites/staging/sqoop/trunk/content/
svn commit: r1630635 - /sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf
Author: jarcec Date: Fri Oct 10 01:12:20 2014 New Revision: 1630635 URL: http://svn.apache.org/r1630635 Log: Fixing extra in the text Modified: sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf Modified: sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf URL: http://svn.apache.org/viewvc/sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf?rev=1630635r1=1630634r2=1630635view=diff == --- sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf (original) +++ sqoop/site/trunk/content/resources/doap/doap_Sqoop.rdf Fri Oct 10 01:12:20 2014 @@ -30,7 +30,7 @@ shortdescApache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases./shortdesc bug-database rdf:resource=http://issues.apache.org/jira/browse/SQOOP; / mailing-list rdf:resource=http://sqoop.apache.org/mail-lists.html; / -download-page rdf:resource=http://sqoop.apache.org/; / +download-page rdf:resource=http://sqoop.apache.org/; / programming-languageJava/programming-language category rdf:resource=http://projects.apache.org/category/big-data; / release
svn commit: r925218 - in /websites/staging/sqoop/trunk/content: ./ index.html issue-tracking.html license.html mail-lists.html project-info.html source-repository.html team-list.html
Author: buildbot Date: Fri Oct 10 01:13:27 2014 New Revision: 925218 Log: Staging update by buildbot for sqoop Modified: websites/staging/sqoop/trunk/content/ (props changed) websites/staging/sqoop/trunk/content/index.html websites/staging/sqoop/trunk/content/issue-tracking.html websites/staging/sqoop/trunk/content/license.html websites/staging/sqoop/trunk/content/mail-lists.html websites/staging/sqoop/trunk/content/project-info.html websites/staging/sqoop/trunk/content/source-repository.html websites/staging/sqoop/trunk/content/team-list.html Propchange: websites/staging/sqoop/trunk/content/ -- --- cms:source-revision (original) +++ cms:source-revision Fri Oct 10 01:13:27 2014 @@ -1 +1 @@ -1630635 +1630636 Modified: websites/staging/sqoop/trunk/content/index.html == --- websites/staging/sqoop/trunk/content/index.html (original) +++ websites/staging/sqoop/trunk/content/index.html Fri Oct 10 01:13:27 2014 @@ -1,13 +1,13 @@ !DOCTYPE html !-- - | Generated by Apache Maven Doxia at 2014-10-09 + | Generated by Apache Maven Doxia at 2014-10-10 | Rendered using Apache Maven Fluido Skin 1.3.0 -- html xmlns=http://www.w3.org/1999/xhtml; xml:lang=en lang=en head meta charset=UTF-8 / meta name=viewport content=width=device-width, initial-scale=1.0 / -meta name=Date-Revision-mmdd content=20141009 / +meta name=Date-Revision-mmdd content=20141010 / meta http-equiv=Content-Language content=en / titleSqoop - /title link rel=stylesheet href=./css/apache-maven-fluido-1.3.0.min.css / @@ -220,7 +220,7 @@ - li id=publishDate class=pull-rightLast Published: 2014-10-09/li + li id=publishDate class=pull-rightLast Published: 2014-10-10/li /ul /div Modified: websites/staging/sqoop/trunk/content/issue-tracking.html == --- websites/staging/sqoop/trunk/content/issue-tracking.html (original) +++ websites/staging/sqoop/trunk/content/issue-tracking.html Fri Oct 10 01:13:27 2014 @@ -1,13 +1,13 @@ !DOCTYPE html !-- - | Generated by Apache Maven Doxia at 2014-10-09 + | Generated by Apache Maven Doxia at 2014-10-10 | Rendered using Apache Maven Fluido Skin 1.3.0 -- html xmlns=http://www.w3.org/1999/xhtml; xml:lang=en lang=en head meta charset=UTF-8 / meta name=viewport content=width=device-width, initial-scale=1.0 / -meta name=Date-Revision-mmdd content=20141009 / +meta name=Date-Revision-mmdd content=20141010 / meta http-equiv=Content-Language content=en / titleSqoop - Issue Tracking/title link rel=stylesheet href=./css/apache-maven-fluido-1.3.0.min.css / @@ -220,7 +220,7 @@ - li id=publishDate class=pull-rightLast Published: 2014-10-09/li + li id=publishDate class=pull-rightLast Published: 2014-10-10/li /ul /div Modified: websites/staging/sqoop/trunk/content/license.html == --- websites/staging/sqoop/trunk/content/license.html (original) +++ websites/staging/sqoop/trunk/content/license.html Fri Oct 10 01:13:27 2014 @@ -1,13 +1,13 @@ !DOCTYPE html !-- - | Generated by Apache Maven Doxia at 2014-10-09 + | Generated by Apache Maven Doxia at 2014-10-10 | Rendered using Apache Maven Fluido Skin 1.3.0 -- html xmlns=http://www.w3.org/1999/xhtml; xml:lang=en lang=en head meta charset=UTF-8 / meta name=viewport content=width=device-width, initial-scale=1.0 / -meta name=Date-Revision-mmdd content=20141009 / +meta name=Date-Revision-mmdd content=20141010 / meta http-equiv=Content-Language content=en / titleSqoop - Project License/title link rel=stylesheet href=./css/apache-maven-fluido-1.3.0.min.css / @@ -220,7 +220,7 @@ - li id=publishDate class=pull-rightLast Published: 2014-10-09/li + li id=publishDate class=pull-rightLast Published: 2014-10-10/li /ul /div Modified: websites/staging/sqoop/trunk/content/mail-lists.html == --- websites/staging/sqoop/trunk/content/mail-lists.html (original) +++ websites/staging/sqoop/trunk/content/mail-lists.html Fri Oct 10 01:13:27 2014 @@ -1,13 +1,13 @@ !DOCTYPE html !-- - | Generated by Apache Maven Doxia at 2014-10-09 + | Generated by Apache Maven Doxia at 2014-10-10 | Rendered using Apache Maven Fluido Skin 1.3.0 -- html xmlns=http
svn commit: r925219 - /websites/staging/sqoop/trunk/content/
Author: buildbot Date: Fri Oct 10 01:14:30 2014 New Revision: 925219 Log: Staging update by buildbot for sqoop Modified: websites/staging/sqoop/trunk/content/ (props changed) Propchange: websites/staging/sqoop/trunk/content/ -- --- cms:source-revision (original) +++ cms:source-revision Fri Oct 10 01:14:30 2014 @@ -1 +1 @@ -1630636 +1630637
svn commit: r925220 - /websites/production/sqoop/content/
Author: jarcec Date: Fri Oct 10 01:15:37 2014 New Revision: 925220 Log: d Added: websites/production/sqoop/content/ - copied from r925219, websites/staging/sqoop/trunk/content/
[40/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
index 5dd7970..39702ca 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
@@ -28,31 +28,38 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.sql.Types;
-import java.util.*;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
-import org.apache.log4j.Logger;
import org.apache.commons.lang.StringUtils;
+import org.apache.log4j.Logger;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.DirectionError;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorHandler;
import org.apache.sqoop.connector.ConnectorManagerUtils;
import org.apache.sqoop.model.MBooleanInput;
-import org.apache.sqoop.model.MLink;
-import org.apache.sqoop.model.MConnectionForms;
-import org.apache.sqoop.model.MEnumInput;
-import org.apache.sqoop.model.MIntegerInput;
-import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MJobForms;
+import org.apache.sqoop.model.MConfig;
+import org.apache.sqoop.model.MConfigType;
import org.apache.sqoop.model.MConnector;
-import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MFormType;
+import org.apache.sqoop.model.MDriver;
import org.apache.sqoop.model.MDriverConfig;
+import org.apache.sqoop.model.MEnumInput;
+import org.apache.sqoop.model.MFromConfig;
import org.apache.sqoop.model.MInput;
import org.apache.sqoop.model.MInputType;
+import org.apache.sqoop.model.MIntegerInput;
+import org.apache.sqoop.model.MJob;
+import org.apache.sqoop.model.MLink;
+import org.apache.sqoop.model.MLinkConfig;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
import org.apache.sqoop.model.MSubmission;
+import org.apache.sqoop.model.MToConfig;
import org.apache.sqoop.repository.JdbcRepositoryContext;
import org.apache.sqoop.repository.JdbcRepositoryHandler;
import org.apache.sqoop.submission.SubmissionStatus;
@@ -93,74 +100,69 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
mc.getUniqueName());
}
mc.setPersistenceId(getConnectorId(mc, conn));
-insertFormsForConnector(mc, conn);
+insertConfigsForConnector(mc, conn);
}
/**
- * Helper method to insert the forms from the into the
- * repository. The job and connector forms within codemc/code will get
- * updated with the id of the forms when this function returns.
- * @param mDriverConfig The MFramework instance to use to upgrade.
- * @param conn JDBC link to use for updating the forms
+ * Helper method to insert the configs from the into the
+ * repository.
+ * @param mDriver The driver instance to use to upgrade.
+ * @param conn JDBC link to use for updating the configs
*/
- private void insertFormsForFramework(MDriverConfig mDriverConfig, Connection
conn) {
-PreparedStatement baseFormStmt = null;
+ private void insertConfigsForDriver(MDriver mDriver, Connection conn) {
+PreparedStatement baseConfigStmt = null;
PreparedStatement baseInputStmt = null;
try{
- baseFormStmt = conn.prepareStatement(STMT_INSERT_FORM_BASE,
+ baseConfigStmt = conn.prepareStatement(STMT_INSERT_CONFIG_BASE,
Statement.RETURN_GENERATED_KEYS);
baseInputStmt = conn.prepareStatement(STMT_INSERT_INPUT_BASE,
Statement.RETURN_GENERATED_KEYS);
- // Register connector forms
- registerForms(null, null, mDriverConfig.getConnectionForms().getForms(),
-MFormType.CONNECTION.name(), baseFormStmt, baseInputStmt);
-
- // Register job forms
- registerForms(null, null, mDriverConfig.getJobForms().getForms(),
-MFormType.JOB.name(), baseFormStmt, baseInputStmt);
+ // Register the job config type, since driver config is per job
+ registerConfigs(null, null, mDriver.getDriverConfig().getConfigs(),
+MConfigType.JOB.name(), baseConfigStmt, baseInputStmt);
} catch (SQLException ex) {
- throw new SqoopException(DerbyRepoError.DERBYREPO_0014,
mDriverConfig.toString(), ex);
+ throw new SqoopException(DerbyRepoError.DERBYREPO_0014,
mDriver.toString(), ex);
} finally {
- closeStatements(baseFormStmt, baseInputStmt);
+ closeStatements(baseConfigStmt, baseInputStmt);
}
}
/**
-
[22/52] [abbrv] SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3d539dd4/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
--
diff --git a/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
b/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
new file mode 100644
index 000..2732b1c
--- /dev/null
+++ b/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
@@ -0,0 +1,173 @@
+package org.apache.sqoop.framework;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.sqoop.common.Direction;
+import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.connector.ConnectorManager;
+import org.apache.sqoop.connector.spi.SqoopConnector;
+import org.apache.sqoop.model.MConnection;
+import org.apache.sqoop.model.MJob;
+import org.apache.sqoop.model.MSubmission;
+import org.apache.sqoop.repository.JdbcRepository;
+import org.apache.sqoop.repository.Repository;
+import org.apache.sqoop.repository.RepositoryManager;
+import org.apache.sqoop.request.HttpEventContext;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestJobManager {
+ private JobManager jobManager;
+ private SqoopConnector sqoopConnectorMock;
+ private ConnectorManager connectorMgrMock;
+ private RepositoryManager repositoryManagerMock;
+ private Repository jdbcRepoMock;
+
+ @Before
+ public void setUp() {
+jobManager = new JobManager();
+connectorMgrMock = mock(ConnectorManager.class);
+sqoopConnectorMock = mock(SqoopConnector.class);
+ConnectorManager.setInstance(connectorMgrMock);
+repositoryManagerMock = mock(RepositoryManager.class);
+RepositoryManager.setInstance(repositoryManagerMock);
+jdbcRepoMock = mock(JdbcRepository.class);
+ }
+
+ @Test
+ public void testCreateJobSubmission() {
+
+HttpEventContext testCtx = new HttpEventContext();
+testCtx.setUsername(testUser);
+MSubmission jobSubmission = jobManager.createJobSubmission(testCtx, 1234L);
+assertEquals(jobSubmission.getCreationUser(), testUser);
+assertEquals(jobSubmission.getLastUpdateUser(), testUser);
+ }
+
+ @Test
+ public void testGetConnector() {
+when(connectorMgrMock.getConnector(123l)).thenReturn(sqoopConnectorMock);
+
when(sqoopConnectorMock.getSupportedDirections()).thenReturn(getSupportedDirections());
+assertEquals(jobManager.getConnector(123l), sqoopConnectorMock);
+verify(connectorMgrMock, times(1)).getConnector(123l);
+ }
+
+ @Test
+ public void testUnsupportedDirectionForConnector() {
+// invalid job id/ direction
+SqoopException exception = new
SqoopException(FrameworkError.FRAMEWORK_0011, Connector:
++ sqoopConnectorMock.getClass().getCanonicalName());
+ListDirection supportedDirections = getSupportedDirections();
+
when(sqoopConnectorMock.getSupportedDirections()).thenReturn(supportedDirections);
+
+try {
+ // invalid direction
+ jobManager.validateSupportedDirection(sqoopConnectorMock, null);
+} catch (SqoopException ex) {
+ assertEquals(ex.getMessage(), exception.getMessage());
+ verify(sqoopConnectorMock, times(1)).getSupportedDirections();
+ return;
+}
+
+fail(Should throw out an exception with message: +
exception.getMessage());
+ }
+
+ @Test
+ public void testGetConnection() {
+MConnection testConnection = new MConnection(123l, null, null);
+testConnection.setEnabled(true);
+MConnection mConnectionSpy = org.mockito.Mockito.spy(testConnection);
+when(repositoryManagerMock.getRepository()).thenReturn(jdbcRepoMock);
+when(jdbcRepoMock.findConnection(123l)).thenReturn(mConnectionSpy);
+assertEquals(jobManager.getConnection(123l), mConnectionSpy);
+verify(repositoryManagerMock, times(1)).getRepository();
+verify(jdbcRepoMock, times(1)).findConnection(123l);
+ }
+
+ @Test
+ public void testDisabledConnection() {
+MConnection testConnection = new MConnection(123l, null, null);
+testConnection.setPersistenceId(1234);
+testConnection.setEnabled(false);
+SqoopException exception = new
SqoopException(FrameworkError.FRAMEWORK_0010, Connection id:
++ testConnection.getPersistenceId());
+
+MConnection mConnectionSpy = org.mockito.Mockito.spy(testConnection);
+when(repositoryManagerMock.getRepository()).thenReturn(jdbcRepoMock);
+when(jdbcRepoMock.findConnection(123l)).thenReturn(mConnectionSpy);
+try {
+ jobManager.getConnection(123l);
+} catch (SqoopException ex) {
+ assertEquals(ex.getMessage(), exception.getMessage());
+ verify(repositoryManagerMock, times(1)).getRepository();
+ verify(jdbcRepoMock,
[46/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/common/src/main/java/org/apache/sqoop/model/MConnector.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnector.java
b/common/src/main/java/org/apache/sqoop/model/MConnector.java
index 7999b08..2f42191 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnector.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnector.java
@@ -23,28 +23,27 @@ import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.common.SupportedDirections;
/**
- * Connector metadata.
- *
- * Includes unique id that identifies connector in metadata store, unique human
- * readable name, corresponding name and all forms for all supported job types.
+ * Connector entity supports the FROM/TO {@link Transferable} Includes unique
id
+ * that identifies connector in the repository, unique human readable name,
+ * corresponding name and all configs to support the from and to data sources
*/
public final class MConnector extends MPersistableEntity implements MClonable {
private final String uniqueName;
private final String className;
- private final MConnectionForms connectionForms;
- private final MJobForms fromJobForms;
- private final MJobForms toJobForms;
- String version;
-
- public MConnector(String uniqueName, String className,
-String version, MConnectionForms connectionForms,
-MJobForms fromJobForms, MJobForms toJobForms) {
+ private final String version;
+ private final MLinkConfig linkConfig;
+ private final MFromConfig fromConfig;
+ private final MToConfig toConfig;
+
+ public MConnector(String uniqueName, String className, String version,
MLinkConfig linkConfig,
+ MFromConfig fromConfig, MToConfig toConfig) {
this.version = version;
-this.connectionForms = connectionForms;
-this.fromJobForms = fromJobForms;
-this.toJobForms = toJobForms;
+this.linkConfig = linkConfig;
+this.fromConfig = fromConfig;
+this.toConfig = toConfig;
+// Why are we abusing NPE?
if (uniqueName == null || className == null) {
throw new NullPointerException();
}
@@ -63,17 +62,15 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
@Override
public String toString() {
-MJobForms fromJobForms = this.getJobForms(Direction.FROM);
-MJobForms toJobForms = this.getJobForms(Direction.TO);
StringBuilder sb = new StringBuilder(connector-);
sb.append(uniqueName).append(:).append(getPersistenceId()).append(:);
sb.append(className);
-sb.append(, ).append(getConnectionForms().toString());
-if (fromJobForms != null) {
- sb.append(, ).append(fromJobForms.toString());
+sb.append(, ).append(getLinkConfig().toString());
+if (getConfig(Direction.FROM) != null) {
+ sb.append(, ).append(getConfig(Direction.FROM).toString());
}
-if (toJobForms != null) {
- sb.append(, ).append(toJobForms.toString());
+if (getConfig(Direction.TO) != null) {
+ sb.append(, ).append(getConfig(Direction.TO).toString());
}
return sb.toString();
}
@@ -94,41 +91,39 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
if (supportedDirections.isDirectionSupported(Direction.FROM)
mcSupportedDirections.isDirectionSupported(Direction.FROM)
-
!getJobForms(Direction.FROM).equals(mc.getJobForms(Direction.FROM))) {
+ !getFromConfig().equals(mc.getFromConfig())) {
return false;
}
-if (supportedDirections.isDirectionSupported(Direction.FROM)
-!= mcSupportedDirections.isDirectionSupported(Direction.FROM)) {
+if (supportedDirections.isDirectionSupported(Direction.FROM) !=
mcSupportedDirections
+.isDirectionSupported(Direction.FROM)) {
return false;
}
if (supportedDirections.isDirectionSupported(Direction.TO)
mcSupportedDirections.isDirectionSupported(Direction.TO)
- !getJobForms(Direction.TO).equals(mc.getJobForms(Direction.TO))) {
+ !getToConfig().equals(mc.getToConfig())) {
return false;
}
-if (supportedDirections.isDirectionSupported(Direction.TO)
-!= mcSupportedDirections.isDirectionSupported(Direction.TO)) {
+if (supportedDirections.isDirectionSupported(Direction.TO) !=
mcSupportedDirections
+.isDirectionSupported(Direction.TO)) {
return false;
}
-return uniqueName.equals(mc.uniqueName)
- className.equals(mc.className)
- version.equals(mc.version)
- connectionForms.equals(mc.getConnectionForms());
+return uniqueName.equals(mc.uniqueName) className.equals(mc.className)
+ version.equals(mc.version)
linkConfig.equals((mc.getLinkConfig()));
}
@Override
public int hashCode() {
SupportedDirections supportedDirections = getSupportedDirections();
-
[35/52] [abbrv] git commit: SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector
SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector (Abraham Elmahrek via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/b6000365 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/b6000365 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/b6000365 Branch: refs/heads/SQOOP-1367 Commit: b6000365736d7b03124417f6b8731d8760011432 Parents: 4c964a9 Author: Jarek Jarcec Cecho jar...@apache.org Authored: Thu Oct 9 08:16:52 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 18:22:51 2014 -0700 -- docs/src/site/sphinx/CommandLineClient.rst | 6 +- docs/src/site/sphinx/ConnectorDevelopment.rst | 154 ++- docs/src/site/sphinx/RESTAPI.rst| 54 +-- .../sqoop/connector/spi/SqoopConnector.java | 2 +- 4 files changed, 95 insertions(+), 121 deletions(-) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/b6000365/docs/src/site/sphinx/CommandLineClient.rst -- diff --git a/docs/src/site/sphinx/CommandLineClient.rst b/docs/src/site/sphinx/CommandLineClient.rst index fb7f780..7cc6449 100644 --- a/docs/src/site/sphinx/CommandLineClient.rst +++ b/docs/src/site/sphinx/CommandLineClient.rst @@ -329,14 +329,14 @@ Create new job object. ++--+ | Argument | Description | ++==+ -| ``-x``, ``--xid x`` | Create new job object for connection with id ``x`` | +| ``-f``, ``--from x`` | Create new job object with a FROM connection with id ``x`` | ++--+ -| ``-t``, ``--type t`` | Create new job object with type ``t`` (``import``, ``export``) | +| ``-t``, ``--to t`` | Create new job object with a TO connection with id ``x`` | ++--+ Example: :: - create job --xid 1 + create job --from 1 --to 2 Update Command -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/b6000365/docs/src/site/sphinx/ConnectorDevelopment.rst -- diff --git a/docs/src/site/sphinx/ConnectorDevelopment.rst b/docs/src/site/sphinx/ConnectorDevelopment.rst index ae4f721..d700e4c 100644 --- a/docs/src/site/sphinx/ConnectorDevelopment.rst +++ b/docs/src/site/sphinx/ConnectorDevelopment.rst @@ -26,17 +26,15 @@ using the code of built-in connector ( ``GenericJdbcConnector`` ) as example. What is Connector? ++ -Connector provides interaction with external databases. -Connector reads data from databases for import, -and write data to databases for export. -Interaction with Hadoop is taken cared by common modules of Sqoop 2 framework. +The connector provides the facilities to interact with external data sources. +The connector can read from, or write to, a data source. When do we add a new connector? === You add a new connector when you need to extract data from a new data source, or load data to a new target. In addition to the connector API, Sqoop 2 also has an engine interface. -At the moment the only engine is MapReduce,but we may support additional engines in the future. +At the moment the only engine is MapReduce, but we may support additional engines in the future. Since many parallel execution engines are capable of reading/writing data there may be a question of whether support for specific data stores should be done through a new connector or new engine. @@ -51,57 +49,73 @@ Connector Implementation The ``SqoopConnector`` class defines functionality which must be provided by Connectors. -Each Connector must extends ``SqoopConnector`` and overrides methods shown below. +Each Connector must extend ``SqoopConnector`` and override the methods shown below. :: public abstract String getVersion(); public abstract ResourceBundle getBundle(Locale locale); - public abstract Class getConnectionConfigurationClass(); - public abstract Class getJobConfigurationClass(MJob.Type jobType); - public abstract Importer getImporter(); - public abstract Exporter getExporter(); + public abstract Class getLinkConfigurationClass(); + public abstract Class getJobConfigurationClass(Direction direction); + public abstract From getFrom(); + public abstract To getTo(); public abstract Validator getValidator(); public abstract MetadataUpgrader
[41/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
--
diff --git
a/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
b/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
index 19b0023..e6e4760 100644
--- a/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
+++ b/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
@@ -17,6 +17,23 @@
*/
package org.apache.sqoop.repository;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyLong;
+import static org.mockito.Matchers.anyObject;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
import java.sql.Connection;
import java.util.ArrayList;
import java.util.Collections;
@@ -29,40 +46,27 @@ import org.apache.sqoop.connector.ConnectorManager;
import org.apache.sqoop.connector.spi.RepositoryUpgrader;
import org.apache.sqoop.connector.spi.SqoopConnector;
import org.apache.sqoop.driver.Driver;
-import org.apache.sqoop.driver.configuration.JobConfiguration;
+import org.apache.sqoop.driver.configuration.DriverConfiguration;
+import org.apache.sqoop.json.DriverBean;
+import org.apache.sqoop.model.ConfigUtils;
import org.apache.sqoop.model.ConfigurationClass;
-import org.apache.sqoop.model.FormUtils;
-import org.apache.sqoop.model.MConnectionForms;
+import org.apache.sqoop.model.MConfig;
+import org.apache.sqoop.model.MConfigList;
import org.apache.sqoop.model.MConnector;
+import org.apache.sqoop.model.MDriver;
import org.apache.sqoop.model.MDriverConfig;
-import org.apache.sqoop.model.MForm;
+import org.apache.sqoop.model.MFromConfig;
import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MJobForms;
import org.apache.sqoop.model.MLink;
+import org.apache.sqoop.model.MLinkConfig;
+import org.apache.sqoop.model.MToConfig;
+import org.apache.sqoop.validation.ConfigValidator;
import org.apache.sqoop.validation.Status;
-import org.apache.sqoop.validation.Validation;
import org.apache.sqoop.validation.Validator;
import org.junit.Before;
import org.junit.Test;
import org.mockito.InOrder;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.fail;
-import static org.mockito.Matchers.any;
-import static org.mockito.Matchers.anyLong;
-import static org.mockito.Matchers.anyObject;
-import static org.mockito.Matchers.anyString;
-import static org.mockito.Mockito.doNothing;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.doThrow;
-import static org.mockito.Mockito.inOrder;
-import static org.mockito.Mockito.mock;
-import static org.mockito.Mockito.spy;
-import static org.mockito.Mockito.times;
-import static org.mockito.Mockito.verify;
-import static org.mockito.Mockito.verifyNoMoreInteractions;
-import static org.mockito.Mockito.when;
-
public class TestJdbcRepository {
private JdbcRepository repoSpy;
@@ -73,8 +77,8 @@ public class TestJdbcRepository {
private Validator validatorMock;
private RepositoryUpgrader upgraderMock;
- private Validation validRepoMock;
- private Validation invalidRepoMock;
+ private ConfigValidator validRepoMock;
+ private ConfigValidator invalidRepoMock;
@Before
public void setUp() throws Exception {
@@ -91,20 +95,20 @@ public class TestJdbcRepository {
ConnectorManager.setInstance(connectorMgrMock);
Driver.setInstance(driverMock);
-validRepoMock = mock(Validation.class);
+validRepoMock = mock(ConfigValidator.class);
when(validRepoMock.getStatus()).thenReturn(Status.ACCEPTABLE);
-invalidRepoMock = mock(Validation.class);
+invalidRepoMock = mock(ConfigValidator.class);
when(invalidRepoMock.getStatus()).thenReturn(Status.UNACCEPTABLE);
-doNothing().when(upgraderMock).upgrade(any(MConnectionForms.class),
any(MConnectionForms.class));
-doNothing().when(upgraderMock).upgrade(any(MJobForms.class),
any(MJobForms.class));
+doNothing().when(upgraderMock).upgrade(any(MLinkConfig.class),
any(MLinkConfig.class));
+doNothing().when(upgraderMock).upgrade(any(MFromConfig.class),
any(MFromConfig.class));
}
/**
* Test the procedure when the connector auto upgrade option is enabled
*/
@Test
- public void testConnectorEnableAutoUpgrade() {
+ public void testConnectorConfigEnableAutoUpgrade() {
MConnector
[06/52] [abbrv] git commit: SQOOP-1465: Sqoop2: Validations: Add support for multiple validation messages to Model classes
SQOOP-1465: Sqoop2: Validations: Add support for multiple validation messages
to Model classes
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/f89368e2
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/f89368e2
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/f89368e2
Branch: refs/heads/SQOOP-1367
Commit: f89368e2a0cc6236b2c50dfe544134f9d55208b9
Parents: e71dd75
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Mon Sep 8 23:16:19 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
.../java/org/apache/sqoop/model/FormUtils.java | 10 +-
.../apache/sqoop/model/MValidatedElement.java | 107 +++
.../org/apache/sqoop/validation/Status.java | 16 ++-
.../org/apache/sqoop/model/TestFormUtils.java | 31 --
.../sqoop/model/TestMValidatedElement.java | 36 +++
.../apache/sqoop/shell/utils/FormDisplayer.java | 5 +-
.../apache/sqoop/shell/utils/FormFiller.java| 17 ++-
7 files changed, 88 insertions(+), 134 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/f89368e2/common/src/main/java/org/apache/sqoop/model/FormUtils.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/FormUtils.java
b/common/src/main/java/org/apache/sqoop/model/FormUtils.java
index bfd97d5..586e0fe 100644
--- a/common/src/main/java/org/apache/sqoop/model/FormUtils.java
+++ b/common/src/main/java/org/apache/sqoop/model/FormUtils.java
@@ -314,9 +314,9 @@ public class FormUtils {
if (messages.containsKey(name)) {
Validation.Message message = messages.get(name);
- element.setValidationMessage(message.getStatus(), message.getMessage());
+ element.addValidationMessage(new Message(message.getStatus(),
message.getMessage()));
} else {
- element.setValidationMessage(Status.getDefault(), null);
+ element.addValidationMessage(new Message(Status.getDefault(), null));
}
}
@@ -349,11 +349,9 @@ public class FormUtils {
ListMessage messages = result.getMessages().get(element.getName());
if(messages != null) {
- // TODO(SQOOP-1465) Add support for multiple messages (showing only the
first one for now)
- Message message = messages.get(0);
- element.setValidationMessage(message.getStatus(), message.getMessage());
+ element.setValidationMessages(messages);
} else {
- element.setValidationMessage(Status.getDefault(), null);
+ element.resetValidationMessages();
}
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/f89368e2/common/src/main/java/org/apache/sqoop/model/MValidatedElement.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MValidatedElement.java
b/common/src/main/java/org/apache/sqoop/model/MValidatedElement.java
index ab5473c..a50c815 100644
--- a/common/src/main/java/org/apache/sqoop/model/MValidatedElement.java
+++ b/common/src/main/java/org/apache/sqoop/model/MValidatedElement.java
@@ -17,122 +17,87 @@
*/
package org.apache.sqoop.model;
+import org.apache.sqoop.validation.Message;
import org.apache.sqoop.validation.Status;
+import java.util.LinkedList;
+import java.util.List;
+
/**
- * Element that can be validated for correctness.
- *
- * Two severity levels are supported at the moment - warning and error.
- *
- * Warning:
- * Warning is something suspicious, potentially wrong but something that
- * can be ignored. For example in case of JDBC URL element, warning would
- * be if specified host is not responding - it's warning because specified
- * URL might be wrong. However at the same time URL might be right as only
- * target host might be down.
- *
- * Error:
- * Error represents unacceptable element content. For example in case of JDBC
- * URL path, error would be empty element or element containing invalid URL.
+ * Element that can have associated validation messages (0..N).
*/
public abstract class MValidatedElement extends MNamedElement {
/**
- * Validation message.
- *
- * One element can have only one message regardless of the type.
+ * Validation messages.
*/
- private String validationMessage;
+ private ListMessage validationMessages;
/**
- * Severity of the message.
+ * The worst status of all validation messages.
*/
private Status validationStatus;
public MValidatedElement(String name) {
super(name);
-// Everything is fine by default
-this.validationStatus = Status.getDefault();
+resetValidationMessages();
}
public MValidatedElement(MValidatedElement other) {
super(other);
-this.validationMessage = other.validationMessage;
[17/52] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
new file mode 100644
index 000..8b23144
--- /dev/null
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.connector.jdbc.configuration;
+
+import org.apache.sqoop.connector.jdbc.GenericJdbcConnectorConstants;
+import org.apache.sqoop.model.FormClass;
+import org.apache.sqoop.model.Input;
+import org.apache.sqoop.model.Validator;
+import org.apache.sqoop.validation.Status;
+import org.apache.sqoop.validation.validators.AbstractValidator;
+import org.apache.sqoop.validation.validators.NullOrContains;
+
+/**
+ *
+ */
+@FormClass( validators = {@Validator(FromJobConfig.FormValidator.class)})
+public class FromJobConfig {
+ @Input(size = 50)
+ public String schemaName;
+
+ @Input(size = 50)
+ public String tableName;
+
+ @Input(size = 2000, validators = {@Validator(value = NullOrContains.class,
strArg = GenericJdbcConnectorConstants.SQL_CONDITIONS_TOKEN)})
+ public String sql;
+
+ @Input(size = 50)
+ public String columns;
+
+ @Input(size = 50)
+ public String partitionColumn;
+
+ @Input
+ public Boolean partitionColumnNull;
+
+ @Input(size = 50)
+ public String boundaryQuery;
+
+ public static class FormValidator extends AbstractValidatorFromJobConfig {
+@Override
+public void validate(FromJobConfig form) {
+ if(form.tableName == null form.sql == null) {
+addMessage(Status.UNACCEPTABLE, Either table name or SQL must be
specified);
+ }
+ if(form.tableName != null form.sql != null) {
+addMessage(Status.UNACCEPTABLE, Both table name and SQL cannot be
specified);
+ }
+ if(form.schemaName != null form.sql != null) {
+addMessage(Status.UNACCEPTABLE, Both schema name and SQL cannot be
specified);
+ }
+}
+ }
+}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
index 64e9a9a..b036421 100644
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
@@ -25,9 +25,9 @@ import org.apache.sqoop.model.Form;
*/
@ConfigurationClass
public class FromJobConfiguration {
- @Form public FromTableForm fromTable;
+ @Form public FromJobConfig fromJobConfig;
public FromJobConfiguration() {
-fromTable = new FromTableForm();
+fromJobConfig = new FromJobConfig();
}
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromTableForm.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromTableForm.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromTableForm.java
deleted file mode 100644
index 72902a2..000
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromTableForm.java
+++ /dev/null
@@ -1,68 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under
[34/52] [abbrv] git commit: SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/fc74316f
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/fc74316f
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/fc74316f
Branch: refs/heads/SQOOP-1367
Commit: fc74316fbdcee80272106064ac4e1e747c300b97
Parents: b600036
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Thu Oct 9 08:36:35 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 18:22:51 2014 -0700
--
.../sqoop/common/SupportedDirections.java | 79 +++-
.../sqoop/common/TestSupportedDirections.java | 19 +
.../sqoop/repository/derby/DerbyRepoError.java | 8 +
.../derby/DerbyRepositoryHandler.java | 398 ---
.../repository/derby/DerbySchemaConstants.java | 53 +++
.../repository/derby/DerbySchemaQuery.java | 164 +++-
.../sqoop/repository/derby/DerbyTestCase.java | 131 --
.../repository/derby/TestConnectorHandling.java | 62 ++-
.../sqoop/repository/derby/TestJobHandling.java | 4 -
.../sqoop/shell/ShowConnectorFunction.java | 30 +-
.../sqoop/connector/spi/SqoopConnector.java | 2 +-
11 files changed, 808 insertions(+), 142 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/fc74316f/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
index 25ba276..c527117 100644
--- a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
+++ b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
@@ -20,7 +20,9 @@ package org.apache.sqoop.common;
/**
* Represents which Directions are supported.
*/
-public class SupportedDirections {
+public class SupportedDirections implements ComparableSupportedDirections {
+ private static final char SUPPORTED_DIRECTIONS_SEPARATOR = '/';
+
private boolean from;
private boolean to;
@@ -38,4 +40,79 @@ public class SupportedDirections {
return direction == Direction.FROM from
|| direction == Direction.TO to;
}
+
+ /**
+ * @return String FROM, TO, FROM/TO, .
+ */
+ public String toString() {
+StringBuffer buffer = new StringBuffer();
+
+if (isDirectionSupported(Direction.FROM)) {
+ buffer.append(Direction.FROM);
+
+ if (isDirectionSupported(Direction.TO)) {
+buffer.append(SUPPORTED_DIRECTIONS_SEPARATOR);
+buffer.append(Direction.TO);
+ }
+} else if (isDirectionSupported(Direction.TO)) {
+ buffer.append(Direction.TO);
+}
+
+return buffer.toString();
+ }
+
+ public static SupportedDirections fromString(String supportedDirections) {
+boolean from = false, to = false;
+
+if (supportedDirections != null !supportedDirections.equals()) {
+ for (String direction : supportedDirections.split(/)) {
+switch (Direction.valueOf(direction)) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+ }
+}
+
+return new SupportedDirections(from, to);
+ }
+
+ public static SupportedDirections fromDirection(Direction direction) {
+boolean from = false, to = false;
+switch (direction) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+return new SupportedDirections(from, to);
+ }
+
+ @Override
+ public int compareTo(SupportedDirections o) {
+int hash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ hash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ hash |= 2;
+}
+
+int oHash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ oHash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ oHash |= 2;
+}
+
+return hash - oHash;
+ }
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/fc74316f/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
--
diff --git
a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
index 4fbaf82..4f0cdd6 100644
--- a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
+++ b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
@@ -52,4 +52,23 @@ public class TestSupportedDirections {
Assert.assertFalse(
[13/52] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/core/src/main/java/org/apache/sqoop/repository/Repository.java
--
diff --git a/core/src/main/java/org/apache/sqoop/repository/Repository.java
b/core/src/main/java/org/apache/sqoop/repository/Repository.java
index 8e8dd80..ea1608a 100644
--- a/core/src/main/java/org/apache/sqoop/repository/Repository.java
+++ b/core/src/main/java/org/apache/sqoop/repository/Repository.java
@@ -17,36 +17,36 @@
*/
package org.apache.sqoop.repository;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
import org.apache.log4j.Logger;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.connector.spi.MetadataUpgrader;
+import org.apache.sqoop.connector.spi.RepositoryUpgrader;
import org.apache.sqoop.connector.spi.SqoopConnector;
-import org.apache.sqoop.framework.FrameworkManager;
+import org.apache.sqoop.driver.Driver;
import org.apache.sqoop.model.FormUtils;
-import org.apache.sqoop.model.MConnection;
import org.apache.sqoop.model.MConnectionForms;
import org.apache.sqoop.model.MConnector;
+import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MFramework;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MJobForms;
+import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MPersistableEntity;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.utils.ClassUtils;
import org.apache.sqoop.validation.Validation;
import org.apache.sqoop.validation.Validator;
-import java.util.Date;
-import java.util.List;
-import java.util.Map;
-
/**
- * Defines the contract of a Repository used by Sqoop. A Repository allows
- * Sqoop to store metadata, statistics and other state relevant to Sqoop
- * Jobs in the system.
+ * Defines the contract for repository used by Sqoop. A Repository allows
+ * Sqoop to store entities such as connectors, links, jobs, submissions and
its related configs,
+ * statistics and other state relevant the entities in the store
*/
public abstract class Repository {
@@ -76,11 +76,11 @@ public abstract class Repository {
/**
* Registers given connector in the repository and return registered
- * variant. This method might return an exception in case that metadata for
+ * variant. This method might return an exception in case that
* given connector are already registered with different structure.
*
- * @param mConnector the connector metadata to be registered
- * autoupgrade whether to upgrade framework automatically
+ * @param mConnector the connector to be registered
+ * autoupgrade whether to upgrade driver config automatically
* @return Registered connector structure
*/
public abstract MConnector registerConnector(MConnector mConnector, boolean
autoUpgrade);
@@ -88,7 +88,7 @@ public abstract class Repository {
/**
* Search for connector with given name in repository.
*
- * And return corresponding metadata structure.
+ * And return corresponding entity structure.
*
* @param shortName Connector unique name
* @return null if connector is not yet registered in repository or
@@ -105,76 +105,75 @@ public abstract class Repository {
/**
- * Registers given framework in the repository and return registered
- * variant. This method might return an exception in case that metadata for
- * given framework are already registered with different structure.
+ * Registers given driverConfig in the repository and return registered
+ * variant. This method might return an exception in case that the
+ * given driverConfig are already registered with different structure.
*
- * @param mFramework framework metadata to be registered
- * autoupgrade whether to upgrade framework automatically
+ * @param mDriverConfig driverConfig to be registered
+ * autoupgrade whether to upgrade driverConfig automatically
* @return Registered connector structure
*/
- public abstract MFramework registerFramework(MFramework mFramework, boolean
autoUpgrade);
+ public abstract MDriverConfig registerDriverConfig(MDriverConfig
mDriverConfig, boolean autoUpgrade);
/**
- * Save given connection to repository. This connection must not be already
+ * Save given link to repository. This link must not be already
* present in the repository otherwise exception will be thrown.
*
- * @param connection Connection object to serialize into repository.
+ * @param link link object to serialize into repository.
*/
- public abstract void createConnection(MConnection connection);
+ public abstract void createLink(MLink link);
/**
- * Update given connection representation in repository. This connection
+ * Update given link representation in
[27/52] [abbrv] git commit: SQOOP-1556: Sqoop2: Add documentation clarifying connectors vs. engines
SQOOP-1556: Sqoop2: Add documentation clarifying connectors vs. engines (Gwen Shapira via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/196346d5 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/196346d5 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/196346d5 Branch: refs/heads/SQOOP-1367 Commit: 196346d5ccbb7cd6f4f2627cbd937728c61301d2 Parents: 35a060e Author: Jarek Jarcec Cecho jar...@apache.org Authored: Fri Sep 26 16:12:22 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 17:59:24 2014 -0700 -- docs/src/site/sphinx/ConnectorDevelopment.rst | 14 ++ 1 file changed, 14 insertions(+) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/196346d5/docs/src/site/sphinx/ConnectorDevelopment.rst -- diff --git a/docs/src/site/sphinx/ConnectorDevelopment.rst b/docs/src/site/sphinx/ConnectorDevelopment.rst index 5121382..ae4f721 100644 --- a/docs/src/site/sphinx/ConnectorDevelopment.rst +++ b/docs/src/site/sphinx/ConnectorDevelopment.rst @@ -31,6 +31,20 @@ Connector reads data from databases for import, and write data to databases for export. Interaction with Hadoop is taken cared by common modules of Sqoop 2 framework. +When do we add a new connector? +=== +You add a new connector when you need to extract data from a new data source, or load +data to a new target. +In addition to the connector API, Sqoop 2 also has an engine interface. +At the moment the only engine is MapReduce,but we may support additional engines in the future. +Since many parallel execution engines are capable of reading/writing data +there may be a question of whether support for specific data stores should be done +through a new connector or new engine. + +**Our guideline is:** Connectors should manage all data extract/load. Engines manage job +life cycles. If you need to support a new data store and don't care how jobs run - +you are looking to add a connector. + Connector Implementation
[09/52] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
--
diff --git
a/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
deleted file mode 100644
index 1eb7e51..000
--- a/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.shell;
-
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.OptionBuilder;
-import org.apache.sqoop.shell.core.Constants;
-import org.apache.sqoop.validation.Status;
-
-import static org.apache.sqoop.shell.ShellEnvironment.*;
-
-/**
- *
- */
-@SuppressWarnings(serial)
-public class DeleteConnectionFunction extends SqoopFunction {
- @SuppressWarnings(static-access)
- public DeleteConnectionFunction() {
-this.addOption(OptionBuilder
- .withDescription(resourceString(Constants.RES_PROMPT_CONN_ID))
- .withLongOpt(Constants.OPT_XID)
- .isRequired()
- .hasArg()
- .create('x'));
- }
-
- @Override
- public Object executeFunction(CommandLine line, boolean isInteractive) {
-client.deleteConnection(getLong(line, Constants.OPT_XID));
-return Status.FINE;
- }
-}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
index 5d48c91..da26153 100644
--- a/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
+++ b/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
@@ -37,7 +37,7 @@ public class DeleteJobFunction extends SqoopFunction {
.withLongOpt(Constants.OPT_JID)
.isRequired()
.hasArg()
- .create('j'));
+ .create(Constants.OPT_JID_CHAR));
}
@Override
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
new file mode 100644
index 000..c81917e
--- /dev/null
+++ b/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.shell;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.sqoop.shell.core.Constants;
+import org.apache.sqoop.validation.Status;
+
+import static org.apache.sqoop.shell.ShellEnvironment.*;
+
+/**
+ *
+ */
+@SuppressWarnings(serial)
+public class DeleteLinkFunction extends SqoopFunction {
+ @SuppressWarnings(static-access)
+ public DeleteLinkFunction() {
+this.addOption(OptionBuilder
+ .withDescription(resourceString(Constants.RES_PROMPT_LINK_ID))
+ .withLongOpt(Constants.OPT_LID)
+ .isRequired()
+ .hasArg()
+ .create(Constants.OPT_LID_CHAR));
+ }
+
+ @Override
+ public Object executeFunction(CommandLine
[50/52] [abbrv] git commit: SQOOP-1477: Sqoop2: Make link and job name unique identifier
SQOOP-1477: Sqoop2: Make link and job name unique identifier
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/4c964a97
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/4c964a97
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/4c964a97
Branch: refs/heads/SQOOP-1367
Commit: 4c964a97be674ead165844274d48fea9ff69fe65
Parents: c78b75a
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Tue Oct 7 11:38:09 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 18:22:51 2014 -0700
--
.../sqoop/repository/RepositoryError.java | 2 +-
.../derby/DerbyRepositoryHandler.java | 18 +++--
.../repository/derby/DerbySchemaConstants.java | 10 +++
.../repository/derby/DerbySchemaQuery.java | 44 +---
.../sqoop/repository/derby/DerbyTestCase.java | 19 --
.../sqoop/repository/derby/TestInternals.java | 71
.../sqoop/repository/derby/TestJobHandling.java | 23 +--
.../repository/derby/TestLinkHandling.java | 12
8 files changed, 156 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/4c964a97/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
--
diff --git
a/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
b/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
index 54e37d9..0c93b50 100644
--- a/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
+++ b/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
@@ -122,7 +122,7 @@ public enum RepositoryError implements ErrorCode {
JDBCREPO_0026(Upgrade required but not allowed),
/** Invalid links or jobs when upgrading connector **/
- JDBCREPO_0027(Invalid links or jobs when upgrading connector);
+ JDBCREPO_0027(Invalid links or jobs when upgrading connector)
;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/4c964a97/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
index 39702ca..73d8387 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
@@ -415,6 +415,10 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
// Wait to remove SQB_TYPE (IMPORT/EXPORT) until we update data.
// Data updates depend on knowledge of the type of job.
runQuery(QUERY_UPGRADE_TABLE_SQ_JOB_REMOVE_COLUMN_SQB_TYPE, conn);
+
+ // Add unique constraints on job and links.
+ runQuery(QUERY_UPGRADE_TABLE_SQ_JOB_ADD_UNIQUE_CONSTRAINT_NAME, conn);
+ runQuery(QUERY_UPGRADE_TABLE_SQ_LINK_ADD_UNIQUE_CONSTRAINT_NAME, conn);
}
ResultSet rs = null;
@@ -849,7 +853,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
PreparedStatement stmt = null;
ResultSet rs = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_LINK_CHECK);
+ stmt = conn.prepareStatement(STMT_SELECT_LINK_CHECK_BY_ID);
stmt.setLong(1, id);
rs = stmt.executeQuery();
@@ -1164,9 +1168,9 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
job.getJobConfig(Direction.TO).getConfigs(),
conn);
createInputValues(STMT_INSERT_JOB_INPUT,
-job.getPersistenceId(),
-job.getDriverConfig().getConfigs(),
-conn);
+ job.getPersistenceId(),
+ job.getDriverConfig().getConfigs(),
+ conn);
} catch (SQLException ex) {
logException(ex, job);
@@ -1184,7 +1188,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
PreparedStatement stmt = null;
ResultSet rs = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_JOB_CHECK);
+ stmt = conn.prepareStatement(STMT_SELECT_JOB_CHECK_BY_ID);
stmt.setLong(1, id);
rs = stmt.executeQuery();
@@ -1279,7 +1283,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
public MJob findJob(long id, Connection conn) {
PreparedStatement stmt = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_JOB_SINGLE);
+ stmt =
[28/52] [abbrv] git commit: SQOOP-1544: Sqoop2: From/To: ConnectorBean should work support different directions
SQOOP-1544: Sqoop2: From/To: ConnectorBean should work support different
directions
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/35a060e0
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/35a060e0
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/35a060e0
Branch: refs/heads/SQOOP-1367
Commit: 35a060e04852c65603223fbb4dc78a18e5c54793
Parents: 2c20d92
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Fri Sep 26 08:03:58 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:59:24 2014 -0700
--
.../org/apache/sqoop/json/ConnectorBean.java| 31 +++
.../apache/sqoop/json/TestConnectorBean.java| 58
.../java/org/apache/sqoop/json/TestUtil.java| 14 -
3 files changed, 92 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/35a060e0/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
--
diff --git a/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
b/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
index 465dd69..d726a14 100644
--- a/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
+++ b/common/src/main/java/org/apache/sqoop/json/ConnectorBean.java
@@ -86,10 +86,15 @@ public class ConnectorBean implements JsonBean {
object.put(CON_FORMS,
extractForms(connector.getConnectionForms().getForms(), skipSensitive));
object.put(JOB_FORMS, new JSONObject());
- ((JSONObject)object.get(JOB_FORMS)).put(
- Direction.FROM,
extractForms(connector.getJobForms(Direction.FROM).getForms(), skipSensitive));
- ((JSONObject)object.get(JOB_FORMS)).put(
- Direction.TO,
extractForms(connector.getJobForms(Direction.TO).getForms(), skipSensitive));
+ if (connector.getJobForms(Direction.FROM) != null) {
+((JSONObject)object.get(JOB_FORMS)).put(
+Direction.FROM,
extractForms(connector.getJobForms(Direction.FROM).getForms(), skipSensitive));
+ }
+
+ if (connector.getJobForms(Direction.TO) != null) {
+((JSONObject)object.get(JOB_FORMS)).put(
+Direction.TO,
extractForms(connector.getJobForms(Direction.TO).getForms(), skipSensitive));
+ }
array.add(object);
}
@@ -124,17 +129,23 @@ public class ConnectorBean implements JsonBean {
String className = (String) object.get(CLASS);
String version = (String) object.get(VERSION);
+ MJobForms fromJob = null;
+ MJobForms toJob = null;
ListMForm connForms = restoreForms((JSONArray) object.get(CON_FORMS));
JSONObject jobJson = (JSONObject) object.get(JOB_FORMS);
JSONArray fromJobJson = (JSONArray)jobJson.get(Direction.FROM.name());
JSONArray toJobJson = (JSONArray)jobJson.get(Direction.TO.name());
- ListMForm fromJobForms = restoreForms(fromJobJson);
- ListMForm toJobForms = restoreForms(toJobJson);
- MJobForms fromJob = new MJobForms(fromJobForms);
- MJobForms toJob = new MJobForms(toJobForms);
+ if (fromJobJson != null) {
+ListMForm fromJobForms = restoreForms(fromJobJson);
+fromJob = new MJobForms(fromJobForms);
+ }
+ if (toJobJson != null) {
+ListMForm toJobForms = restoreForms(toJobJson);
+toJob = new MJobForms(toJobForms);
+ }
MConnectionForms connection = new MConnectionForms(connForms);
- MConnector connector = new MConnector(uniqueName, className, version,
connection, fromJob,
- toJob);
+ MConnector connector = new MConnector(uniqueName, className, version,
+ connection, fromJob, toJob);
connector.setPersistenceId(connectorId);
connectors.add(connector);
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/35a060e0/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
--
diff --git a/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
b/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
index e078474..c3aa84f 100644
--- a/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
+++ b/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
@@ -73,4 +73,62 @@ public class TestConnectorBean {
assertEquals(a, retrievedBundle.getString(a));
assertEquals(b, retrievedBundle.getString(b));
}
+
+ @Test
+ public void testSingleDirection() {
+// Create testing connector
+ListMConnector connectors = new LinkedListMConnector();
+connectors.add(getConnector(jdbc, true, false));
+connectors.add(getConnector(mysql, false, true));
+
+// Create testing bundles
+
[04/52] [abbrv] git commit: SQOOP-1487: Sqoop2: From/To: Refactor/Create HDFS connector test cases
SQOOP-1487: Sqoop2: From/To: Refactor/Create HDFS connector test cases
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/c0b22b1d
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/c0b22b1d
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/c0b22b1d
Branch: refs/heads/SQOOP-1367
Commit: c0b22b1d62221f2a24294520b8ff3851c06d06c3
Parents: 8fee134
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Wed Sep 3 09:40:11 2014 +0200
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
connector/connector-hdfs/pom.xml| 5 +
.../sqoop/connector/hdfs/HdfsExtractor.java | 2 -
.../apache/sqoop/connector/hdfs/FileUtils.java | 82 ++
.../sqoop/connector/hdfs/TestExtractor.java | 125 ++
.../sqoop/connector/hdfs/TestHdfsBase.java | 139 +++
.../apache/sqoop/connector/hdfs/TestLoader.java | 213
.../sqoop/connector/hdfs/TestPartitioner.java | 113 +
.../src/test/resources/log4j.properties | 24 ++
.../java/org/apache/sqoop/job/FileUtils.java| 69 -
.../org/apache/sqoop/job/TestHdfsExtract.java | 241 --
.../java/org/apache/sqoop/job/TestHdfsLoad.java | 250 ---
11 files changed, 701 insertions(+), 562 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/pom.xml
--
diff --git a/connector/connector-hdfs/pom.xml b/connector/connector-hdfs/pom.xml
index 8df9f11..fa4330a 100644
--- a/connector/connector-hdfs/pom.xml
+++ b/connector/connector-hdfs/pom.xml
@@ -35,6 +35,11 @@ limitations under the License.
dependencies
dependency
+ groupIdjunit/groupId
+ artifactIdjunit/artifactId
+/dependency
+
+dependency
groupIdorg.apache.sqoop/groupId
artifactIdsqoop-spi/artifactId
/dependency
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
--
diff --git
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
index fc12381..7447071 100644
---
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
+++
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
@@ -42,8 +42,6 @@ import java.io.IOException;
* Extract from HDFS.
* Default field delimiter of a record is comma.
*/
-
-
public class HdfsExtractor extends ExtractorConnectionConfiguration,
FromJobConfiguration, HdfsPartition {
public static final Logger LOG = Logger.getLogger(HdfsExtractor.class);
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
--
diff --git
a/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
b/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
new file mode 100644
index 000..8c19d01
--- /dev/null
+++
b/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.connector.hdfs;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.LinkedList;
+import java.util.List;
+
+public class FileUtils {
+
+ public static boolean exists(String file) throws IOException {
+Path
[30/52] [abbrv] SQOOP-1478: Sqoop2: Migrate remaining tests from junit 3 to 4
http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
index 7378eda..8dd1ce2 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
@@ -22,11 +22,15 @@ import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MForm;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
+import org.junit.Before;
+import org.junit.Test;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import static org.junit.Assert.*;
+
/**
* Test link methods on Derby repository.
*/
@@ -34,7 +38,7 @@ public class TestLinkHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
- @Override
+ @Before
public void setUp() throws Exception {
super.setUp();
@@ -47,6 +51,7 @@ public class TestLinkHandling extends DerbyTestCase {
loadConnectorAndDriverConfig();
}
+ @Test
public void testFindLink() throws Exception {
// Let's try to find non existing link
try {
@@ -81,6 +86,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertNull(forms.get(1).getInputs().get(1).getValue());
}
+ @Test
public void testFindLinks() throws Exception {
ListMLink list;
@@ -98,6 +104,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(CB, list.get(1).getName());
}
+ @Test
public void testExistsLink() throws Exception {
// There shouldn't be anything on empty repository
assertFalse(handler.existsLink(1, getDerbyDatabaseConnection()));
@@ -111,6 +118,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertFalse(handler.existsLink(3, getDerbyDatabaseConnection()));
}
+ @Test
public void testCreateLink() throws Exception {
MLink link = getLink();
@@ -150,6 +158,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertCountForTable(SQOOP.SQ_CONNECTION_INPUT, 8);
}
+ @Test
public void testInUseLink() throws Exception {
loadLinks();
@@ -160,6 +169,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertTrue(handler.inUseLink(1, getDerbyDatabaseConnection()));
}
+ @Test
public void testUpdateLink() throws Exception {
loadLinks();
@@ -205,6 +215,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(((Map)forms.get(1).getInputs().get(1).getValue()).size(), 0);
}
+ @Test
public void testEnableAndDisableLink() throws Exception {
loadLinks();
@@ -223,6 +234,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(true, retrieved.getEnabled());
}
+ @Test
public void testDeleteLink() throws Exception {
loadLinks();
http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
index 38a995f..cf0944d 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
@@ -22,11 +22,15 @@ import org.apache.sqoop.submission.SubmissionStatus;
import org.apache.sqoop.submission.counter.Counter;
import org.apache.sqoop.submission.counter.CounterGroup;
import org.apache.sqoop.submission.counter.Counters;
+import org.junit.Before;
+import org.junit.Test;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
+import static org.junit.Assert.*;
+
/**
*
*/
@@ -34,7 +38,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
- @Override
+ @Before
public void setUp() throws Exception {
super.setUp();
@@ -53,6 +57,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
loadJobs();
}
+ @Test
public void testFindSubmissionsUnfinished() throws Exception {
ListMSubmission submissions;
@@ -67,6 +72,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
assertEquals(2, submissions.size());
}
+ @Test
public void testExistsSubmission() throws Exception {
// There shouldn't be anything on empty repository
[36/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
b/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
deleted file mode 100644
index a0a5dc2..000
--- a/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
+++ /dev/null
@@ -1,939 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.shell.utils;
-
-import jline.ConsoleReader;
-
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.lang.StringUtils;
-import org.apache.sqoop.common.Direction;
-import org.apache.sqoop.model.MBooleanInput;
-import org.apache.sqoop.model.MLink;
-import org.apache.sqoop.model.MEnumInput;
-import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MInput;
-import org.apache.sqoop.model.MIntegerInput;
-import org.apache.sqoop.model.MMapInput;
-import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MNamedElement;
-import org.apache.sqoop.model.MStringInput;
-import org.apache.sqoop.model.MValidatedElement;
-import org.apache.sqoop.validation.Message;
-import org.apache.sqoop.validation.Status;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-import java.util.HashMap;
-import java.util.ResourceBundle;
-
-import static org.apache.sqoop.shell.ShellEnvironment.*;
-
-/**
- * Convenient methods for retrieving user input and CLI options.
- */
-public final class FormFiller {
-
- /**
- * Internal input that will be reused for loading names for link and
- * job objects.
- */
- private static MStringInput nameInput = new MStringInput(object-name,
false, (short)25);
-
- /**
- * Fill job object based on CLI options.
- *
- * @param line Associated console reader object
- * @param job Job that user is suppose to fill in
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillJob(CommandLine line,
-MJob job)
-throws IOException {
-
-job.setName(line.getOptionValue(name));
-
-// Fill in data from user
-return fillForms(line,
- job.getConnectorPart(Direction.FROM).getForms(),
- job.getFrameworkPart().getForms());
- }
-
- /**
- * Fill job object based on user input.
- *
- * @param reader Associated console reader object
- * @param job Job that user is suppose to fill in
- * @param fromConnectorBundle Connector resource bundle
- * @param driverConfigBundle Driver config resource bundle
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillJob(ConsoleReader reader,
-MJob job,
-ResourceBundle fromConnectorBundle,
-ResourceBundle driverConfigBundle,
-ResourceBundle toConnectorBundle)
-throws IOException {
-
-job.setName(getName(reader, job.getName()));
-
-// Fill in data from user
-return fillForms(reader,
- job.getConnectorPart(Direction.FROM).getForms(),
- fromConnectorBundle,
- job.getFrameworkPart().getForms(),
- driverConfigBundle,
- job.getConnectorPart(Direction.TO).getForms(),
- toConnectorBundle);
- }
-
- /**
- * Fill link object based on CLI options.
- *
- * @param line Associated command line options
- * @param link Link that user is suppose to fill in
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillConnection(CommandLine line,
- MLink link)
- throws IOException {
-
-link.setName(line.getOptionValue(name));
-
-// Fill in data from user
-return
[52/52] [abbrv] git commit: SQOOP-1574: Sqoop2: From/To: Rebase against Sqoop2 branch
SQOOP-1574: Sqoop2: From/To: Rebase against Sqoop2 branch
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/d7e04904
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/d7e04904
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/d7e04904
Branch: refs/heads/SQOOP-1367
Commit: d7e04904300ddef91cb70d65efbc16493b206815
Parents: fc8eb9b
Author: Abraham Elmahrek abra...@elmahrek.com
Authored: Thu Oct 9 19:06:32 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 19:39:41 2014 -0700
--
.../apache/sqoop/tools/tool/JSONConstants.java | 4 +-
.../sqoop/tools/tool/RepositoryDumpTool.java| 15 +-
.../sqoop/tools/tool/RepositoryLoadTool.java| 190 +--
3 files changed, 95 insertions(+), 114 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/d7e04904/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
--
diff --git a/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
b/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
index 288cba3..9bec0d0 100644
--- a/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
+++ b/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
@@ -28,9 +28,9 @@ public final class JSONConstants {
public static final String CONNECTOR_NAME = connector-name;
public static final String ALL = all;
public static final String NAME = name;
- public static final String CONNECTION_ID = connection-id;
+ public static final String LINK_ID = link-id;
public static final String JOB_ID = job-id;
- public static final String CONNECTIONS = connections;
+ public static final String LINKS = links;
public static final String JOBS = jobs;
public static final String SUBMISSIONS = submissions;
public static final String METADATA = metadata;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/d7e04904/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
--
diff --git
a/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
b/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
index d41b0d2..f89c546 100644
--- a/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
+++ b/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
@@ -25,25 +25,21 @@ import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.ParseException;
import org.apache.log4j.Logger;
import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.json.ConnectionBean;
import org.apache.sqoop.json.JobBean;
+import org.apache.sqoop.json.LinkBean;
import org.apache.sqoop.json.SubmissionBean;
-import org.apache.sqoop.model.MConnector;
import org.apache.sqoop.repository.Repository;
import org.apache.sqoop.repository.RepositoryManager;
import org.apache.sqoop.tools.ConfiguredTool;
import org.apache.sqoop.common.VersionInfo;
-import static org.apache.sqoop.json.util.FormSerialization.ALL;
+import static org.apache.sqoop.json.util.ConfigSerialization.ALL;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
-import java.util.HashMap;
import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
/**
* Write user-created content of Sqoop repository to JSON formatted file
@@ -106,8 +102,8 @@ public class RepositoryDumpTool extends ConfiguredTool {
JSONObject result = new JSONObject();
LOG.info(Dumping Connections with skipSensitive= +
String.valueOf(skipSensitive));
-ConnectionBean connections = new
ConnectionBean(repository.findConnections());
-result.put(JSONConstants.CONNECTIONS,
addConnectorName(connections.extract(skipSensitive)));
+LinkBean links = new LinkBean(repository.findLinks());
+result.put(JSONConstants.LINKS,
addConnectorName(links.extract(skipSensitive)));
LOG.info(Dumping Jobs with skipSensitive= +
String.valueOf(skipSensitive));
JobBean jobs = new JobBean(repository.findJobs());
@@ -134,7 +130,6 @@ public class RepositoryDumpTool extends ConfiguredTool {
}
private JSONObject addConnectorName(JSONObject json) {
-Repository repository = RepositoryManager.getInstance().getRepository();
ConnectorManager connectorManager = ConnectorManager.getInstance();
JSONArray results = (JSONArray) json.get(ALL);
@@ -144,7 +139,7 @@ public class RepositoryDumpTool extends ConfiguredTool {
while (iterator.hasNext()) {
JSONObject result = iterator.next();
Long connectorId = (Long)
[39/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
index ad42901..951d9b4 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
@@ -48,17 +48,17 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_FORM/strong: Form details.
+ * strongSQ_CONFIG/strong: Config details.
* pre
*+--+
- *| SQ_FORM |
+ *| SQ_CONFIG |
*+--+
- *| SQF_ID: BIGINT PK AUTO-GEN |
- *| SQF_CONNECTOR: BIGINT| FK SQ_CONNECTOR(SQC_ID),NULL for
framework
- *| SQF_DIRECTION: VARCHAR(32) | FROM|TO|NULL
- *| SQF_NAME: VARCHAR(64)|
- *| SQF_TYPE: VARCHAR(32)| CONNECTION|JOB
- *| SQF_INDEX: SMALLINT |
+ *| SQ_CFG_ID: BIGINT PK AUTO-GEN |
+ *| SQ_CFG_OWNER: BIGINT| FK SQ_CFG_OWNER(SQC_ID),NULL for
driver
+ *| SQ_CFG_DIRECTION: VARCHAR(32) | FROM|TO|NULL
+ *| SQ_CFG_NAME: VARCHAR(64)|
+ *| SQ_CFG_TYPE: VARCHAR(32)| LINK|JOB
+ *| SQ_CFG_INDEX: SMALLINT |
*+--+
* /pre
* /p
@@ -70,7 +70,7 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
*++
*| SQI_ID: BIGINT PK AUTO-GEN |
*| SQI_NAME: VARCHAR(64) |
- *| SQI_FORM: BIGINT | FK SQ_FORM(SQF_ID)
+ *| SQI_CONFIG: BIGINT | FK SQ_CONFIG(SQ_CFG_ID)
*| SQI_INDEX: SMALLINT|
*| SQI_TYPE: VARCHAR(32) | STRING|MAP
*| SQI_STRMASK: BOOLEAN |
@@ -80,19 +80,19 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_CONNECTION/strong: Stored connections
+ * strongSQ_LINK/strong: Stored connections
* pre
*++
- *| SQ_CONNECTION |
+ *| SQ_LINK |
*++
- *| SQN_ID: BIGINT PK AUTO-GEN |
- *| SQN_NAME: VARCHAR(64) |
- *| SQN_CONNECTOR: BIGINT | FK SQ_CONNECTOR(SQC_ID)
- *| SQN_CREATION_USER: VARCHAR(32) |
- *| SQN_CREATION_DATE: TIMESTAMP |
- *| SQN_UPDATE_USER: VARCHAR(32) |
- *| SQN_UPDATE_DATE: TIMESTAMP |
- *| SQN_ENABLED: BOOLEAN |
+ *| SQ_LNK_ID: BIGINT PK AUTO-GEN |
+ *| SQ_LNK_NAME: VARCHAR(64) |
+ *| SQ_LNK_CONNECTOR: BIGINT | FK SQ_CONNECTOR(SQC_ID)
+ *| SQ_LNK_CREATION_USER: VARCHAR(32) |
+ *| SQ_LNK_CREATION_DATE: TIMESTAMP |
+ *| SQ_LNK_UPDATE_USER: VARCHAR(32) |
+ *| SQ_LNK_UPDATE_DATE: TIMESTAMP |
+ *| SQ_LNK_ENABLED: BOOLEAN |
*++
* /pre
* /p
@@ -104,8 +104,8 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
*++
*| SQB_ID: BIGINT PK AUTO-GEN |
*| SQB_NAME: VARCHAR(64) |
- *| SQB_FROM_CONNECTION: BIGINT| FK SQ_CONNECTION(SQN_ID)
- *| SQB_TO_CONNECTION: BIGINT | FK SQ_CONNECTION(SQN_ID)
+ *| SQB_FROM_LINK: BIGINT| FK SQ_LINK(SQ_LNK_ID)
+ *| SQB_TO_LINK: BIGINT | FK SQ_LINK(SQ_LNK_ID)
*| SQB_CREATION_USER: VARCHAR(32) |
*| SQB_CREATION_DATE: TIMESTAMP |
*| SQB_UPDATE_USER: VARCHAR(32) |
@@ -115,14 +115,14 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_CONNECTION_INPUT/strong: N:M relationship connection and input
+ * strongSQ_LINK_INPUT/strong: N:M relationship link and input
* pre
*++
- *| SQ_CONNECTION_INPUT|
+ *| SQ_LINK_INPUT|
*++
- *| SQNI_CONNECTION: BIGINT PK | FK SQ_CONNECTION(SQN_ID)
- *| SQNI_INPUT: BIGINT PK | FK SQ_INPUT(SQI_ID)
- *| SQNI_VALUE: LONG VARCHAR |
+ *| SQ_LNKI_LINK: BIGINT PK | FK SQ_LINK(SQ_LNK_ID)
+ *| SQ_LNKI_INPUT: BIGINT PK | FK SQ_INPUT(SQI_ID)
+ *| SQ_LNKI_VALUE: LONG VARCHAR |
*++
* /pre
* /p
@@ -221,17 +221,17 @@ public final class DerbySchemaQuery {
+ COLUMN_SQC_VERSION + VARCHAR(64)
+ );
- // DDL: Create table SQ_FORM
- public static final
[20/52] [abbrv] git commit: SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
SQOOP-1497: Sqoop2: Entity Nomenclature Revisited Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/049994a0 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/049994a0 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/049994a0 Branch: refs/heads/SQOOP-1367 Commit: 049994a0217164b703e754978a85a6c17fcb720a Parents: b04e796 Author: Veena Basavaraj veena.b...@gmail.com Authored: Tue Sep 23 15:28:24 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 17:58:18 2014 -0700 -- .../org/apache/sqoop/client/SqoopClient.java| 280 .../sqoop/client/request/ConnectionRequest.java | 91 --- .../sqoop/client/request/ConnectorRequest.java | 47 -- .../request/ConnectorResourceRequest.java | 47 ++ .../request/DriverConfigResourceRequest.java| 42 ++ .../sqoop/client/request/FrameworkRequest.java | 42 -- .../apache/sqoop/client/request/JobRequest.java | 91 --- .../client/request/JobResourceRequest.java | 91 +++ .../client/request/LinkResourceRequest.java | 89 +++ .../apache/sqoop/client/request/Request.java| 110 --- .../sqoop/client/request/ResourceRequest.java | 113 +++ .../sqoop/client/request/SqoopRequests.java | 149 .../client/request/SqoopResourceRequests.java | 149 .../sqoop/client/request/SubmissionRequest.java | 80 --- .../request/SubmissionResourceRequest.java | 80 +++ .../sqoop/client/request/VersionRequest.java| 35 - .../client/request/VersionResourceRequest.java | 35 + .../apache/sqoop/client/TestSqoopClient.java| 105 ++- .../org/apache/sqoop/common/SqoopException.java | 2 +- .../org/apache/sqoop/etl/io/DataReader.java | 10 +- .../org/apache/sqoop/job/etl/ActorContext.java | 77 -- .../apache/sqoop/job/etl/DestroyerContext.java | 2 +- .../apache/sqoop/job/etl/ExtractorContext.java | 2 +- .../sqoop/job/etl/InitializerContext.java | 2 +- .../org/apache/sqoop/job/etl/LoaderContext.java | 2 +- .../sqoop/job/etl/PartitionerContext.java | 2 +- .../sqoop/job/etl/TransferableContext.java | 76 ++ .../org/apache/sqoop/json/ConnectionBean.java | 185 - .../sqoop/json/ConnectionValidationBean.java| 143 .../org/apache/sqoop/json/ConnectorBean.java| 37 +- .../org/apache/sqoop/json/DriverConfigBean.java | 92 +++ .../org/apache/sqoop/json/FrameworkBean.java| 102 --- .../java/org/apache/sqoop/json/JobBean.java | 63 +- .../java/org/apache/sqoop/json/LinkBean.java| 184 + .../apache/sqoop/json/LinkValidationBean.java | 143 .../sqoop/json/util/FormSerialization.java | 2 +- .../json/util/ResourceBundleSerialization.java | 13 +- .../main/java/org/apache/sqoop/model/Input.java | 2 +- .../org/apache/sqoop/model/MConnection.java | 130 .../org/apache/sqoop/model/MDriverConfig.java | 97 +++ .../java/org/apache/sqoop/model/MFramework.java | 99 --- .../java/org/apache/sqoop/model/MInput.java | 2 +- .../main/java/org/apache/sqoop/model/MJob.java | 34 +- .../main/java/org/apache/sqoop/model/MLink.java | 129 .../apache/sqoop/json/TestConnectionBean.java | 138 .../apache/sqoop/json/TestDriverConfigBean.java | 64 ++ .../apache/sqoop/json/TestFrameworkBean.java| 64 -- .../java/org/apache/sqoop/json/TestJobBean.java | 4 +- .../org/apache/sqoop/json/TestLinkBean.java | 138 .../java/org/apache/sqoop/json/TestUtil.java| 23 +- .../apache/sqoop/json/TestValidationBean.java | 12 +- .../sqoop/model/TestMAccountableEntity.java | 24 +- .../org/apache/sqoop/model/TestMConnection.java | 122 .../apache/sqoop/model/TestMDriverConfig.java | 47 ++ .../org/apache/sqoop/model/TestMFramework.java | 47 -- .../java/org/apache/sqoop/model/TestMLink.java | 122 .../connector/jdbc/GenericJdbcConnector.java| 12 +- .../jdbc/GenericJdbcConnectorConstants.java | 2 +- .../GenericJdbcConnectorMetadataUpgrader.java | 83 --- .../jdbc/GenericJdbcConnectorUpgrader.java | 83 +++ .../connector/jdbc/GenericJdbcExtractor.java| 15 +- .../jdbc/GenericJdbcFromDestroyer.java | 6 +- .../jdbc/GenericJdbcFromInitializer.java| 60 +- .../sqoop/connector/jdbc/GenericJdbcLoader.java | 14 +- .../connector/jdbc/GenericJdbcPartitioner.java | 8 +- .../connector/jdbc/GenericJdbcToDestroyer.java | 22 +- .../jdbc/GenericJdbcToInitializer.java | 52 +- .../connector/jdbc/GenericJdbcValidator.java| 66 +- .../configuration/ConnectionConfiguration.java | 34 - .../jdbc/configuration/ConnectionForm.java | 64 -- .../jdbc/configuration/FromJobConfig.java | 68 ++ .../configuration/FromJobConfiguration.java | 4 +- .../jdbc/configuration/FromTableForm.java | 68 -- .../jdbc/configuration/LinkConfig.java | 64 ++
[31/52] [abbrv] git commit: SQOOP-1478: Sqoop2: Migrate remaining tests from junit 3 to 4
SQOOP-1478: Sqoop2: Migrate remaining tests from junit 3 to 4 Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/97da12aa Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/97da12aa Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/97da12aa Branch: refs/heads/SQOOP-1367 Commit: 97da12aa5043e1d2f1fcfaf2531670c13aa0c2ae Parents: 196346d Author: Chiwan Park chiwanp...@icloud.com Authored: Mon Sep 29 11:04:48 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 17:59:24 2014 -0700 -- .../org/apache/sqoop/common/TestMapContext.java | 3 +- .../apache/sqoop/json/TestConnectorBean.java| 3 +- .../apache/sqoop/json/TestDriverConfigBean.java | 2 -- .../java/org/apache/sqoop/json/TestJobBean.java | 2 +- .../org/apache/sqoop/json/TestLinkBean.java | 2 +- .../org/apache/sqoop/json/TestSchemaBean.java | 2 -- .../apache/sqoop/json/TestSubmissionBean.java | 30 ++--- .../sqoop/json/TestValidationResultBean.java| 4 +-- .../org/apache/sqoop/model/TestFormUtils.java | 12 +-- .../sqoop/model/TestMAccountableEntity.java | 4 +-- .../apache/sqoop/model/TestMBooleanInput.java | 1 - .../sqoop/model/TestMConnectionForms.java | 4 +-- .../org/apache/sqoop/model/TestMConnector.java | 4 +-- .../org/apache/sqoop/model/TestMEnumInput.java | 4 +-- .../java/org/apache/sqoop/model/TestMForm.java | 3 +- .../org/apache/sqoop/model/TestMFormList.java | 2 +- .../apache/sqoop/model/TestMIntegerInput.java | 4 +-- .../java/org/apache/sqoop/model/TestMJob.java | 4 +-- .../org/apache/sqoop/model/TestMJobForms.java | 4 +-- .../java/org/apache/sqoop/model/TestMLink.java | 4 +-- .../org/apache/sqoop/model/TestMMapInput.java | 10 +++--- .../apache/sqoop/model/TestMNamedElement.java | 4 +-- .../sqoop/model/TestMPersistableEntity.java | 3 +- .../apache/sqoop/model/TestMStringInput.java| 4 +-- .../sqoop/model/TestMValidatedElement.java | 4 +-- .../sqoop/submission/TestSubmissionStatus.java | 10 -- .../submission/counter/TestCounterGroup.java| 2 +- .../sqoop/submission/counter/TestCounters.java | 2 +- .../org/apache/sqoop/utils/TestClassUtils.java | 1 - .../org/apache/sqoop/validation/TestStatus.java | 2 +- .../apache/sqoop/validation/TestValidation.java | 8 +++-- .../validation/validators/TestContains.java | 2 +- .../validators/TestNullOrContains.java | 2 +- .../validation/validators/TestStartsWith.java | 2 +- .../connector/jdbc/GenericJdbcExecutorTest.java | 12 +-- .../sqoop/connector/jdbc/TestExtractor.java | 20 .../connector/jdbc/TestFromInitializer.java | 24 ++ .../apache/sqoop/connector/jdbc/TestLoader.java | 6 ++-- .../sqoop/connector/jdbc/TestPartitioner.java | 25 +++--- .../sqoop/connector/jdbc/TestToInitializer.java | 26 --- .../org/apache/sqoop/driver/TestJobManager.java | 14 .../org/apache/sqoop/driver/TestJobRequest.java | 6 ++-- .../sqoop/repository/TestJdbcRepository.java| 34 ++-- .../java/org/apache/sqoop/job/JobUtils.java | 2 +- .../org/apache/sqoop/job/TestMapReduce.java | 15 + .../apache/sqoop/job/io/SqoopWritableTest.java | 10 -- .../java/org/apache/sqoop/job/io/TestData.java | 9 +++--- .../mr/TestSqoopOutputFormatLoadExecutor.java | 2 +- .../sqoop/repository/derby/DerbyTestCase.java | 16 - .../repository/derby/TestConnectorHandling.java | 11 ++- .../derby/TestDriverConfigHandling.java | 11 ++- .../sqoop/repository/derby/TestInputTypes.java | 10 +- .../sqoop/repository/derby/TestInternals.java | 11 ++- .../sqoop/repository/derby/TestJobHandling.java | 14 +++- .../repository/derby/TestLinkHandling.java | 14 +++- .../derby/TestSubmissionHandling.java | 12 ++- 56 files changed, 310 insertions(+), 148 deletions(-) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/common/src/test/java/org/apache/sqoop/common/TestMapContext.java -- diff --git a/common/src/test/java/org/apache/sqoop/common/TestMapContext.java b/common/src/test/java/org/apache/sqoop/common/TestMapContext.java index f4718c0..7ce1ccd 100644 --- a/common/src/test/java/org/apache/sqoop/common/TestMapContext.java +++ b/common/src/test/java/org/apache/sqoop/common/TestMapContext.java @@ -20,8 +20,7 @@ package org.apache.sqoop.common; import java.util.HashMap; import java.util.Map; -import junit.framework.Assert; - +import org.junit.Assert; import org.junit.Test; /** http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/common/src/test/java/org/apache/sqoop/json/TestConnectorBean.java
[24/52] [abbrv] git commit: SQOOP-1454: Sqoop2: From/To: Add client support for supported directions
SQOOP-1454: Sqoop2: From/To: Add client support for supported directions
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/27fb31d4
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/27fb31d4
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/27fb31d4
Branch: refs/heads/SQOOP-1367
Commit: 27fb31d42e7e76ca14def568e454121437f4e9e5
Parents: 3d539dd
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Sun Sep 21 12:42:10 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../sqoop/common/SupportedDirections.java | 41 ++
.../java/org/apache/sqoop/model/MConnector.java | 6 +
.../sqoop/common/TestSupportedDirections.java | 55
.../org/apache/sqoop/model/TestMConnector.java | 127 ---
.../sqoop/connector/ConnectorHandler.java | 4 -
.../sqoop/shell/ShowConnectorFunction.java | 37 +-
.../org/apache/sqoop/shell/core/Constants.java | 2 +
.../main/resources/shell-resource.properties| 5 +-
8 files changed, 226 insertions(+), 51 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
new file mode 100644
index 000..25ba276
--- /dev/null
+++ b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.common;
+
+/**
+ * Represents which Directions are supported.
+ */
+public class SupportedDirections {
+ private boolean from;
+ private boolean to;
+
+ public SupportedDirections(boolean from, boolean to) {
+this.from = from;
+this.to = to;
+ }
+
+ /**
+ * Check if direction is supported.
+ * @param direction
+ * @return boolean
+ */
+ public boolean isDirectionSupported(Direction direction) {
+return direction == Direction.FROM from
+|| direction == Direction.TO to;
+ }
+}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/main/java/org/apache/sqoop/model/MConnector.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnector.java
b/common/src/main/java/org/apache/sqoop/model/MConnector.java
index 335a0cc..3dc1014 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnector.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnector.java
@@ -20,6 +20,7 @@ package org.apache.sqoop.model;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.DirectionError;
import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.common.SupportedDirections;
/**
* Connector metadata.
@@ -139,4 +140,9 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
public void setVersion(String version) {
this.version = version;
}
+
+ public SupportedDirections getSupportedDirections() {
+return new SupportedDirections(this.getJobForms(Direction.FROM) != null,
+this.getJobForms(Direction.TO) != null);
+ }
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
--
diff --git
a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
new file mode 100644
index 000..4fbaf82
--- /dev/null
+++ b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE
[38/52] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
index 47350ea..595b1c8 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
@@ -17,9 +17,14 @@
*/
package org.apache.sqoop.repository.derby;
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.model.MForm;
+import org.apache.sqoop.model.MConfig;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
@@ -38,18 +43,19 @@ import static org.junit.Assert.*;
public class TestJobHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
+ Connection derbyConnection;
@Before
public void setUp() throws Exception {
super.setUp();
+derbyConnection = getDerbyDatabaseConnection();
handler = new DerbyRepositoryHandler();
// We always needs schema for this test case
createSchema();
-// We always needs connector and framework structures in place
-loadConnectorAndDriverConfig();
+loadConnectorLinkConfig();
// We always needs connection metadata in place
loadLinks();
@@ -59,58 +65,53 @@ public class TestJobHandling extends DerbyTestCase {
public void testFindJob() throws Exception {
// Let's try to find non existing job
try {
- handler.findJob(1, getDerbyDatabaseConnection());
+ handler.findJob(1, derbyConnection);
fail();
} catch(SqoopException ex) {
assertEquals(DerbyRepoError.DERBYREPO_0030, ex.getErrorCode());
}
-// Load prepared connections into database
loadJobs();
-MJob jobImport = handler.findJob(1, getDerbyDatabaseConnection());
-assertNotNull(jobImport);
-assertEquals(1, jobImport.getPersistenceId());
-assertEquals(JA, jobImport.getName());
-
-ListMForm forms;
-
-// Check connector parts
-forms = jobImport.getConnectorPart(Direction.FROM).getForms();
-assertEquals(2, forms.size());
-assertEquals(Value5, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value5, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
-
-forms = jobImport.getConnectorPart(Direction.TO).getForms();
-assertEquals(2, forms.size());
-assertEquals(Value9, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value9, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
-
-// Check framework part
-forms = jobImport.getFrameworkPart().getForms();
-assertEquals(2, forms.size());
-assertEquals(Value17, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value19, forms.get(1).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
+MJob firstJob = handler.findJob(1, derbyConnection);
+assertNotNull(firstJob);
+assertEquals(1, firstJob.getPersistenceId());
+assertEquals(JA, firstJob.getName());
+
+ListMConfig configs;
+
+configs = firstJob.getJobConfig(Direction.FROM).getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value5, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value5, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
+
+configs = firstJob.getJobConfig(Direction.TO).getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value9, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value9, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
+
+configs = firstJob.getDriverConfig().getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value13, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value15, configs.get(1).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
}
@Test
public void
[48/52] [abbrv] git commit: SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M) (Veena Basavaraj via Abraham Elmahrek) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/8362c73c Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/8362c73c Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/8362c73c Branch: refs/heads/SQOOP-1367 Commit: 8362c73cc01313d2510a659bf296efd727bb1d4e Parents: e540668 Author: Abraham Elmahrek abra...@elmahrek.com Authored: Thu Oct 2 23:29:50 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 18:22:51 2014 -0700 -- .../org/apache/sqoop/client/SqoopClient.java| 119 +-- .../request/ConnectorResourceRequest.java | 4 +- .../request/DriverConfigResourceRequest.java| 15 +- .../client/request/JobResourceRequest.java | 26 +- .../client/request/LinkResourceRequest.java | 10 +- .../client/request/SqoopResourceRequests.java | 4 +- .../apache/sqoop/client/TestSqoopClient.java| 73 +- .../org/apache/sqoop/json/ConnectorBean.java| 129 +-- .../java/org/apache/sqoop/json/DriverBean.java | 90 ++ .../org/apache/sqoop/json/DriverConfigBean.java | 92 -- .../java/org/apache/sqoop/json/JobBean.java | 77 +- .../apache/sqoop/json/JobValidationBean.java| 83 +- .../java/org/apache/sqoop/json/LinkBean.java| 96 +- .../apache/sqoop/json/LinkValidationBean.java | 56 +- .../org/apache/sqoop/json/ThrowableBean.java| 2 +- .../apache/sqoop/json/ValidationResultBean.java | 18 +- .../sqoop/json/util/ConfigSerialization.java| 226 + .../sqoop/json/util/FormSerialization.java | 223 - .../json/util/ResourceBundleSerialization.java | 2 - .../java/org/apache/sqoop/model/Config.java | 34 + .../org/apache/sqoop/model/ConfigClass.java | 45 + .../org/apache/sqoop/model/ConfigUtils.java | 637 + .../apache/sqoop/model/ConfigurationClass.java | 7 +- .../main/java/org/apache/sqoop/model/Form.java | 35 - .../java/org/apache/sqoop/model/FormClass.java | 45 - .../java/org/apache/sqoop/model/FormUtils.java | 634 - .../java/org/apache/sqoop/model/MConfig.java| 117 +++ .../org/apache/sqoop/model/MConfigList.java | 124 +++ .../org/apache/sqoop/model/MConfigType.java | 34 + .../apache/sqoop/model/MConnectionForms.java| 54 -- .../java/org/apache/sqoop/model/MConnector.java | 124 ++- .../java/org/apache/sqoop/model/MDriver.java| 82 ++ .../org/apache/sqoop/model/MDriverConfig.java | 65 +- .../main/java/org/apache/sqoop/model/MForm.java | 117 --- .../java/org/apache/sqoop/model/MFormList.java | 124 --- .../java/org/apache/sqoop/model/MFormType.java | 34 - .../org/apache/sqoop/model/MFromConfig.java | 64 ++ .../main/java/org/apache/sqoop/model/MJob.java | 102 +- .../java/org/apache/sqoop/model/MJobForms.java | 55 -- .../main/java/org/apache/sqoop/model/MLink.java | 72 +- .../org/apache/sqoop/model/MLinkConfig.java | 54 ++ .../java/org/apache/sqoop/model/MToConfig.java | 64 ++ .../java/org/apache/sqoop/model/ModelError.java | 6 +- .../java/org/apache/sqoop/utils/ClassUtils.java | 6 +- .../sqoop/validation/ConfigValidationError.java | 52 + .../validation/ConfigValidationResult.java | 98 ++ .../validation/ConfigValidationRunner.java | 177 .../sqoop/validation/ConfigValidator.java | 228 + .../org/apache/sqoop/validation/Message.java| 2 +- .../org/apache/sqoop/validation/Validation.java | 228 - .../sqoop/validation/ValidationError.java | 52 - .../sqoop/validation/ValidationResult.java | 98 -- .../sqoop/validation/ValidationRunner.java | 176 .../org/apache/sqoop/json/ConfigTestUtil.java | 217 + .../apache/sqoop/json/TestConnectorBean.java| 37 +- .../org/apache/sqoop/json/TestDriverBean.java | 62 ++ .../apache/sqoop/json/TestDriverConfigBean.java | 62 -- .../java/org/apache/sqoop/json/TestJobBean.java | 40 +- .../org/apache/sqoop/json/TestLinkBean.java | 56 +- .../java/org/apache/sqoop/json/TestUtil.java| 153 --- .../apache/sqoop/json/TestValidationBean.java | 83 +- .../sqoop/json/TestValidationResultBean.java| 24 +- .../json/util/TestConfigSerialization.java | 157 .../sqoop/json/util/TestFormSerialization.java | 157 .../org/apache/sqoop/model/TestConfigUtils.java | 290 ++ .../org/apache/sqoop/model/TestFormUtils.java | 295 -- .../sqoop/model/TestMAccountableEntity.java | 9 +- .../org/apache/sqoop/model/TestMConfig.java | 86 ++ .../org/apache/sqoop/model/TestMConfigList.java | 55 ++ .../sqoop/model/TestMConnectionForms.java | 48 - .../org/apache/sqoop/model/TestMConnector.java | 144 +-- .../org/apache/sqoop/model/TestMDriver.java | 40 + .../apache/sqoop/model/TestMDriverConfig.java |
[25/52] [abbrv] git commit: SQOOP-1488: Sqoop2: From/To: Run both destroyers
SQOOP-1488: Sqoop2: From/To: Run both destroyers
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/b04e796f
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/b04e796f
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/b04e796f
Branch: refs/heads/SQOOP-1367
Commit: b04e796f01cb659efc55314029fa18cfd80bb16d
Parents: 27fb31d
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Sun Sep 21 13:00:46 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../mapreduce/MapreduceExecutionEngine.java | 3 +-
.../java/org/apache/sqoop/job/JobConstants.java | 6 ++--
.../sqoop/job/mr/SqoopDestroyerExecutor.java| 33 ++--
.../sqoop/job/mr/SqoopFileOutputFormat.java | 7 +++--
.../sqoop/job/mr/SqoopNullOutputFormat.java | 7 +++--
.../org/apache/sqoop/job/TestMapReduce.java | 29 +
.../jdbc/generic/TableStagedRDBMSTest.java | 15 -
7 files changed, 75 insertions(+), 25 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/b04e796f/execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
--
diff --git
a/execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
b/execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
index ef7ff4e..049d183 100644
---
a/execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
+++
b/execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
@@ -68,7 +68,8 @@ public class MapreduceExecutionEngine extends ExecutionEngine
{
context.setString(JobConstants.JOB_ETL_PARTITIONER,
from.getPartitioner().getName());
context.setString(JobConstants.JOB_ETL_EXTRACTOR,
from.getExtractor().getName());
context.setString(JobConstants.JOB_ETL_LOADER, to.getLoader().getName());
-context.setString(JobConstants.JOB_ETL_DESTROYER,
from.getDestroyer().getName());
+context.setString(JobConstants.JOB_ETL_FROM_DESTROYER,
from.getDestroyer().getName());
+context.setString(JobConstants.JOB_ETL_TO_DESTROYER,
to.getDestroyer().getName());
context.setString(JobConstants.INTERMEDIATE_DATA_FORMAT,
mrJobRequest.getIntermediateDataFormat().getName());
http://git-wip-us.apache.org/repos/asf/sqoop/blob/b04e796f/execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
--
diff --git
a/execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
b/execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
index 4cdb002..542607a 100644
--- a/execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
+++ b/execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java
@@ -37,9 +37,11 @@ public final class JobConstants extends Constants {
public static final String JOB_ETL_LOADER = PREFIX_JOB_CONFIG
+ etl.loader;
- public static final String JOB_ETL_DESTROYER = PREFIX_JOB_CONFIG
- + etl.destroyer;
+ public static final String JOB_ETL_FROM_DESTROYER = PREFIX_JOB_CONFIG
+ + etl.from.destroyer;
+ public static final String JOB_ETL_TO_DESTROYER = PREFIX_JOB_CONFIG
+ + etl.to.destroyer;
public static final String JOB_MR_OUTPUT_FILE = PREFIX_JOB_CONFIG
+ mr.output.file;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/b04e796f/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
--
diff --git
a/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
b/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
index e3af6e1..aecde40 100644
---
a/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
+++
b/execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
@@ -40,10 +40,24 @@ public class SqoopDestroyerExecutor {
* @param success True if the job execution was successfull
* @param configuration Configuration object to get destroyer class with
context
* and configuration objects.
- * @param propertyName Name of property that holds destroyer class.
+ * @param direction The direction of the Destroyer to execute.
*/
- public static void executeDestroyer(boolean success, Configuration
configuration, String propertyName) {
-Destroyer destroyer = (Destroyer)
[01/52] [abbrv] SQOOP-1374: From/To: Metadata upgrade
Repository: sqoop
Updated Branches:
refs/heads/SQOOP-1367 0cfcce5aa - d7e049043 (forced update)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/51a07bc3/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
index 2260a45..e658c11 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
@@ -17,6 +17,7 @@
*/
package org.apache.sqoop.repository.derby;
+import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.model.MForm;
import org.apache.sqoop.model.MJob;
@@ -32,242 +33,249 @@ import java.util.Map;
*/
public class TestJobHandling extends DerbyTestCase {
-// DerbyRepositoryHandler handler;
-//
-// @Override
-// public void setUp() throws Exception {
-//super.setUp();
-//
-//handler = new DerbyRepositoryHandler();
-//
-//// We always needs schema for this test case
-//createSchema();
-//
-//// We always needs connector and framework structures in place
-//loadConnectorAndFramework();
-//
-//// We always needs connection metadata in place
-//loadConnections();
-// }
-//
-// public void testFindJob() throws Exception {
-//// Let's try to find non existing job
-//try {
-// handler.findJob(1, getDerbyConnection());
-// fail();
-//} catch(SqoopException ex) {
-// assertEquals(DerbyRepoError.DERBYREPO_0030, ex.getErrorCode());
-//}
-//
-//// Load prepared connections into database
-//loadJobs();
-//
-//MJob jobImport = handler.findJob(1, getDerbyConnection());
-//assertNotNull(jobImport);
-//assertEquals(1, jobImport.getPersistenceId());
-//assertEquals(JA, jobImport.getName());
-//assertEquals(MJob.Type.IMPORT, jobImport.getType());
-//
-//ListMForm forms;
-//
-//// Check connector part
-//forms = jobImport.getFromPart().getForms();
-//assertEquals(Value5, forms.get(0).getInputs().get(0).getValue());
-//assertNull(forms.get(0).getInputs().get(1).getValue());
-//assertEquals(Value7, forms.get(1).getInputs().get(0).getValue());
-//assertNull(forms.get(1).getInputs().get(1).getValue());
-//
-//// Check framework part
-//forms = jobImport.getFrameworkPart().getForms();
-//assertEquals(Value17, forms.get(0).getInputs().get(0).getValue());
-//assertNull(forms.get(0).getInputs().get(1).getValue());
-//assertEquals(Value19, forms.get(1).getInputs().get(0).getValue());
-//assertNull(forms.get(1).getInputs().get(1).getValue());
-// }
-//
-// public void testFindJobs() throws Exception {
-//ListMJob list;
-//
-//// Load empty list on empty repository
-//list = handler.findJobs(getDerbyConnection());
-//assertEquals(0, list.size());
-//
-//loadJobs();
-//
-//// Load all two connections on loaded repository
-//list = handler.findJobs(getDerbyConnection());
-//assertEquals(4, list.size());
-//
-//assertEquals(JA, list.get(0).getName());
-//assertEquals(MJob.Type.IMPORT, list.get(0).getType());
-//
-//assertEquals(JB, list.get(1).getName());
-//assertEquals(MJob.Type.IMPORT, list.get(1).getType());
-//
-//assertEquals(JA, list.get(2).getName());
-//assertEquals(MJob.Type.EXPORT, list.get(2).getType());
-//
-//assertEquals(JB, list.get(3).getName());
-//assertEquals(MJob.Type.EXPORT, list.get(3).getType());
-// }
-//
-// public void testExistsJob() throws Exception {
-//// There shouldn't be anything on empty repository
-//assertFalse(handler.existsJob(1, getDerbyConnection()));
-//assertFalse(handler.existsJob(2, getDerbyConnection()));
-//assertFalse(handler.existsJob(3, getDerbyConnection()));
-//assertFalse(handler.existsJob(4, getDerbyConnection()));
-//assertFalse(handler.existsJob(5, getDerbyConnection()));
-//
-//loadJobs();
-//
-//assertTrue(handler.existsJob(1, getDerbyConnection()));
-//assertTrue(handler.existsJob(2, getDerbyConnection()));
-//assertTrue(handler.existsJob(3, getDerbyConnection()));
-//assertTrue(handler.existsJob(4, getDerbyConnection()));
-//assertFalse(handler.existsJob(5, getDerbyConnection()));
-// }
-//
-// public void testInUseJob() throws Exception {
-//loadJobs();
-//loadSubmissions();
-//
-//assertTrue(handler.inUseJob(1, getDerbyConnection()));
-//assertFalse(handler.inUseJob(2, getDerbyConnection()));
-//assertFalse(handler.inUseJob(3, getDerbyConnection()));
-//assertFalse(handler.inUseJob(4, getDerbyConnection()));
-// }
-//
-// public void
[07/52] [abbrv] git commit: SQOOP-1542: Sqoop2: From/To: MConnector should handle null forms
SQOOP-1542: Sqoop2: From/To: MConnector should handle null forms
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/d0153621
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/d0153621
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/d0153621
Branch: refs/heads/SQOOP-1367
Commit: d0153621d38c4c8478276417b12e36bd67ed6121
Parents: 049994a
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Wed Sep 24 16:16:54 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../java/org/apache/sqoop/model/MConnector.java | 64 +---
.../org/apache/sqoop/model/TestMConnector.java | 52
2 files changed, 107 insertions(+), 9 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/d0153621/common/src/main/java/org/apache/sqoop/model/MConnector.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnector.java
b/common/src/main/java/org/apache/sqoop/model/MConnector.java
index 3dc1014..7999b08 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnector.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnector.java
@@ -63,12 +63,18 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
@Override
public String toString() {
+MJobForms fromJobForms = this.getJobForms(Direction.FROM);
+MJobForms toJobForms = this.getJobForms(Direction.TO);
StringBuilder sb = new StringBuilder(connector-);
sb.append(uniqueName).append(:).append(getPersistenceId()).append(:);
sb.append(className);
sb.append(, ).append(getConnectionForms().toString());
-sb.append(, ).append(getJobForms(Direction.FROM).toString());
-sb.append(, ).append(getJobForms(Direction.TO).toString());
+if (fromJobForms != null) {
+ sb.append(, ).append(fromJobForms.toString());
+}
+if (toJobForms != null) {
+ sb.append(, ).append(toJobForms.toString());
+}
return sb.toString();
}
@@ -83,19 +89,47 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
}
MConnector mc = (MConnector) other;
+SupportedDirections supportedDirections = this.getSupportedDirections();
+SupportedDirections mcSupportedDirections = mc.getSupportedDirections();
+
+if (supportedDirections.isDirectionSupported(Direction.FROM)
+ mcSupportedDirections.isDirectionSupported(Direction.FROM)
+
!getJobForms(Direction.FROM).equals(mc.getJobForms(Direction.FROM))) {
+ return false;
+}
+
+if (supportedDirections.isDirectionSupported(Direction.FROM)
+!= mcSupportedDirections.isDirectionSupported(Direction.FROM)) {
+ return false;
+}
+
+if (supportedDirections.isDirectionSupported(Direction.TO)
+ mcSupportedDirections.isDirectionSupported(Direction.TO)
+ !getJobForms(Direction.TO).equals(mc.getJobForms(Direction.TO))) {
+ return false;
+}
+
+if (supportedDirections.isDirectionSupported(Direction.TO)
+!= mcSupportedDirections.isDirectionSupported(Direction.TO)) {
+ return false;
+}
+
return uniqueName.equals(mc.uniqueName)
className.equals(mc.className)
version.equals(mc.version)
- connectionForms.equals(mc.getConnectionForms())
- fromJobForms.equals(mc.getJobForms(Direction.FROM))
- toJobForms.equals(mc.getJobForms(Direction.TO));
+ connectionForms.equals(mc.getConnectionForms());
}
@Override
public int hashCode() {
+SupportedDirections supportedDirections = getSupportedDirections();
int result = getConnectionForms().hashCode();
-result = 31 * result + getJobForms(Direction.FROM).hashCode();
-result = 31 * result + getJobForms(Direction.TO).hashCode();
+if (supportedDirections.isDirectionSupported(Direction.FROM)) {
+ result = 31 * result + getJobForms(Direction.FROM).hashCode();
+}
+if (supportedDirections.isDirectionSupported(Direction.TO)) {
+ result = 31 * result + getJobForms(Direction.TO).hashCode();
+}
result = 31 * result + version.hashCode();
result = 31 * result + uniqueName.hashCode();
result = 31 * result + className.hashCode();
@@ -105,13 +139,25 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
public MConnector clone(boolean cloneWithValue) {
//Connector never have any values filled
cloneWithValue = false;
+
+MJobForms fromJobForms = this.getJobForms(Direction.FROM);
+MJobForms toJobForms = this.getJobForms(Direction.TO);
+
+if (fromJobForms != null) {
+ fromJobForms = fromJobForms.clone(cloneWithValue);
+}
[03/52] [abbrv] git commit: SQOOP-1374: From/To: Metadata upgrade
SQOOP-1374: From/To: Metadata upgrade
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/51a07bc3
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/51a07bc3
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/51a07bc3
Branch: refs/heads/SQOOP-1367
Commit: 51a07bc352dff37e9482744d272fc54112a1861c
Parents: cd882a9
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Mon Sep 1 15:15:53 2014 +0200
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
.../org/apache/sqoop/model/MConnection.java |1 +
.../main/java/org/apache/sqoop/model/MJob.java |1 +
.../GenericJdbcConnectorMetadataUpgrader.java |8 +-
.../connector/jdbc/GenericJdbcValidator.java|4 +-
.../sqoop/connector/hdfs/HdfsConnector.java |2 +-
.../connector/hdfs/HdfsMetadataUpgrader.java| 83 ++
.../sqoop/connector/ConnectorManager.java | 53 +-
.../sqoop/connector/ConnectorManagerUtils.java | 70 ++
.../apache/sqoop/repository/JdbcRepository.java | 12 +-
.../org/apache/sqoop/repository/Repository.java | 54 +-
.../TestFrameworkMetadataUpgrader.java | 270 ++---
.../repository/derby/DerbyRepoConstants.java|4 +-
.../derby/DerbyRepositoryHandler.java | 258 -
.../repository/derby/DerbySchemaConstants.java | 14 +
.../repository/derby/DerbySchemaQuery.java | 218 +++-
.../sqoop/repository/derby/DerbyTestCase.java | 1084 +++---
.../derby/TestConnectionHandling.java | 418 +++
.../repository/derby/TestConnectorHandling.java | 132 +--
.../repository/derby/TestFrameworkHandling.java | 193 ++--
.../sqoop/repository/derby/TestInputTypes.java | 206 ++--
.../sqoop/repository/derby/TestInternals.java | 62 +-
.../sqoop/repository/derby/TestJobHandling.java | 484
.../derby/TestSubmissionHandling.java | 420 +++
.../sqoop/connector/spi/MetadataUpgrader.java |3 +-
24 files changed, 2386 insertions(+), 1668 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/51a07bc3/common/src/main/java/org/apache/sqoop/model/MConnection.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnection.java
b/common/src/main/java/org/apache/sqoop/model/MConnection.java
index e5a4fb8..f84abbf 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnection.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnection.java
@@ -66,6 +66,7 @@ public class MConnection extends MAccountableEntity
implements MClonable {
this.connectorId = other.connectorId;
this.connectorPart = connectorPart;
this.frameworkPart = frameworkPart;
+this.setPersistenceId(other.getPersistenceId());
}
@Override
http://git-wip-us.apache.org/repos/asf/sqoop/blob/51a07bc3/common/src/main/java/org/apache/sqoop/model/MJob.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MJob.java
b/common/src/main/java/org/apache/sqoop/model/MJob.java
index 11839fc..182bbfb 100644
--- a/common/src/main/java/org/apache/sqoop/model/MJob.java
+++ b/common/src/main/java/org/apache/sqoop/model/MJob.java
@@ -106,6 +106,7 @@ public class MJob extends MAccountableEntity implements
MClonable {
this.fromConnectorPart = fromPart;
this.toConnectorPart = toPart;
this.frameworkPart = frameworkPart;
+this.setPersistenceId(other.getPersistenceId());
}
@Override
http://git-wip-us.apache.org/repos/asf/sqoop/blob/51a07bc3/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
index 2b12009..cbe72f6 100644
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcConnectorMetadataUpgrader.java
@@ -49,7 +49,6 @@ public class GenericJdbcConnectorMetadataUpgrader extends
MetadataUpgrader {
@Override
public void upgrade(MJobForms original, MJobForms upgradeTarget) {
doUpgrade(original.getForms(), upgradeTarget.getForms());
-
}
@SuppressWarnings(unchecked)
@@ -65,12 +64,17 @@ public class GenericJdbcConnectorMetadataUpgrader extends
MetadataUpgrader {
for (MForm form
[16/52] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsMetadataUpgrader.java
--
diff --git
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsMetadataUpgrader.java
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsMetadataUpgrader.java
deleted file mode 100644
index 3e51e38..000
---
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsMetadataUpgrader.java
+++ /dev/null
@@ -1,83 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * AS IS BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.sqoop.connector.hdfs;
-
-import org.apache.log4j.Logger;
-import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.connector.spi.MetadataUpgrader;
-import org.apache.sqoop.model.MConnectionForms;
-import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MInput;
-import org.apache.sqoop.model.MJobForms;
-
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-public class HdfsMetadataUpgrader extends MetadataUpgrader {
- private static final Logger LOG =
- Logger.getLogger(HdfsMetadataUpgrader.class);
-
- /*
- * For now, there is no real upgrade. So copy all data over,
- * set the validation messages and error messages to be the same as for the
- * inputs in the original one.
- */
-
- @Override
- public void upgrade(MConnectionForms original,
- MConnectionForms upgradeTarget) {
-doUpgrade(original.getForms(), upgradeTarget.getForms());
- }
-
- @Override
- public void upgrade(MJobForms original, MJobForms upgradeTarget) {
-doUpgrade(original.getForms(), upgradeTarget.getForms());
- }
-
- @SuppressWarnings(unchecked)
- private void doUpgrade(ListMForm original, ListMForm target) {
-// Easier to find the form in the original forms list if we use a map.
-// Since the constructor of MJobForms takes a list,
-// index is not guaranteed to be the same, so we need to look for
-// equivalence
-MapString, MForm formMap = new HashMapString, MForm();
-for (MForm form : original) {
- formMap.put(form.getName(), form);
-}
-for (MForm form : target) {
- ListMInput? inputs = form.getInputs();
- MForm originalForm = formMap.get(form.getName());
- if (originalForm == null) {
-LOG.warn(Form: ' + form.getName() + ' not present in old +
-connector. So it and its inputs will not be transferred by the
upgrader.);
-continue;
- }
- for (MInput input : inputs) {
-try {
- MInput originalInput = originalForm.getInput(input.getName());
- input.setValue(originalInput.getValue());
-} catch (SqoopException ex) {
- LOG.warn(Input: ' + input.getName() + ' not present in old +
- connector. So it will not be transferred by the upgrader.);
-}
- }
-}
- }
-}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
--
diff --git
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
index df764d2..6828de8 100644
---
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
+++
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsPartitioner.java
@@ -39,7 +39,7 @@ import org.apache.hadoop.io.compress.CompressionCodecFactory;
import org.apache.hadoop.net.NodeBase;
import org.apache.hadoop.net.NetworkTopology;
import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.connector.hdfs.configuration.ConnectionConfiguration;
+import org.apache.sqoop.connector.hdfs.configuration.LinkConfiguration;
import org.apache.sqoop.connector.hdfs.configuration.FromJobConfiguration;
import org.apache.sqoop.job.etl.Partition;
import org.apache.sqoop.job.etl.Partitioner;
@@ -50,7 +50,7 @@ import
[05/52] [abbrv] git commit: SQOOP-1468: Sqoop2: Validations: Introduce advanced validators that can be parametrized
SQOOP-1468: Sqoop2: Validations: Introduce advanced validators that can be
parametrized
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/e71dd757
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/e71dd757
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/e71dd757
Branch: refs/heads/SQOOP-1367
Commit: e71dd75793982b1d6e5109b8ef2209c835a79b68
Parents: c0b22b1
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Wed Sep 3 11:17:48 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
.../apache/sqoop/model/ConfigurationClass.java | 4 +-
.../java/org/apache/sqoop/model/FormClass.java | 4 +-
.../main/java/org/apache/sqoop/model/Input.java | 4 +-
.../java/org/apache/sqoop/model/Validator.java | 45
.../sqoop/validation/ValidationResult.java | 4 +-
.../sqoop/validation/ValidationRunner.java | 18 ++--
.../validators/AbstractValidator.java | 103 +++
.../validation/validators/ClassAvailable.java | 2 +-
.../sqoop/validation/validators/Contains.java | 32 ++
.../sqoop/validation/validators/NotEmpty.java | 2 +-
.../sqoop/validation/validators/NotNull.java| 2 +-
.../sqoop/validation/validators/StartsWith.java | 32 ++
.../sqoop/validation/validators/Validator.java | 80 --
.../sqoop/validation/TestValidationRunner.java | 42 ++--
.../validators/TestClassAvailable.java | 2 +-
.../validation/validators/TestContains.java | 90
.../validation/validators/TestNotEmpty.java | 2 +-
.../validation/validators/TestNotNull.java | 2 +-
.../validation/validators/TestStartsWith.java | 74 +
.../validation/validators/TestValidator.java| 2 +-
.../jdbc/configuration/ConnectionForm.java | 12 ++-
.../jdbc/configuration/FromTableForm.java | 19 ++--
.../jdbc/configuration/ToTableForm.java | 7 +-
23 files changed, 450 insertions(+), 134 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
b/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
index 5323bd9..73374d8 100644
--- a/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
+++ b/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -37,5 +35,5 @@ public @interface ConfigurationClass {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/FormClass.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/FormClass.java
b/common/src/main/java/org/apache/sqoop/model/FormClass.java
index 48bff3c..6048d03 100644
--- a/common/src/main/java/org/apache/sqoop/model/FormClass.java
+++ b/common/src/main/java/org/apache/sqoop/model/FormClass.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -43,5 +41,5 @@ public @interface FormClass {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/Input.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/Input.java
b/common/src/main/java/org/apache/sqoop/model/Input.java
index 61fc01a..2b32e43 100644
--- a/common/src/main/java/org/apache/sqoop/model/Input.java
+++ b/common/src/main/java/org/apache/sqoop/model/Input.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -51,5 +49,5 @@ public @interface Input {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
[29/52] [abbrv] git commit: SQOOP-1560: Sqoop2: Move matcher out of Schema
SQOOP-1560: Sqoop2: Move matcher out of Schema
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/c1e53e5d
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/c1e53e5d
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/c1e53e5d
Branch: refs/heads/SQOOP-1367
Commit: c1e53e5d54e3e9453aefe74b2dd171ff9b25e5a3
Parents: 97da12a
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Thu Oct 2 16:55:29 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:59:24 2014 -0700
--
.../idf/CSVIntermediateDataFormat.java | 134 -
.../connector/idf/IntermediateDataFormat.java | 9 +-
.../connector/idf/matcher/AbstractMatcher.java | 62 -
.../connector/idf/matcher/LocationMatcher.java | 82 --
.../connector/idf/matcher/NameMatcher.java | 69 -
.../connector/matcher/LocationMatcher.java | 78 ++
.../apache/sqoop/connector/matcher/Matcher.java | 69 +
.../sqoop/connector/matcher/MatcherError.java | 41 +++
.../sqoop/connector/matcher/MatcherFactory.java | 30 ++
.../sqoop/connector/matcher/NameMatcher.java| 74 +
.../idf/TestCSVIntermediateDataFormat.java | 160 ++-
.../org/apache/sqoop/job/mr/SqoopMapper.java| 43 +--
.../job/mr/SqoopOutputFormatLoadExecutor.java | 16 +-
.../java/org/apache/sqoop/job/JobUtils.java | 12 +-
.../org/apache/sqoop/job/TestMapReduce.java | 12 +-
.../java/org/apache/sqoop/job/TestMatching.java | 275 +++
16 files changed, 681 insertions(+), 485 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c1e53e5d/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
--
diff --git
a/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
b/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
index 2a49221..02d1a51 100644
---
a/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
+++
b/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
@@ -22,12 +22,7 @@ import com.google.common.annotations.VisibleForTesting;
import org.apache.commons.lang.StringUtils;
import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.connector.idf.matcher.AbstractMatcher;
-import org.apache.sqoop.connector.idf.matcher.LocationMatcher;
-import org.apache.sqoop.connector.idf.matcher.NameMatcher;
import org.apache.sqoop.schema.Schema;
-import org.apache.sqoop.schema.SchemaError;
-import org.apache.sqoop.schema.SchemaMatchOption;
import org.apache.sqoop.schema.type.Column;
import org.apache.sqoop.schema.type.FixedPoint;
import org.apache.sqoop.schema.type.FloatingPoint;
@@ -41,7 +36,6 @@ import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
-import java.util.Set;
import java.util.regex.Matcher;
public class CSVIntermediateDataFormat extends IntermediateDataFormatString {
@@ -71,8 +65,7 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
private final ListInteger stringFieldIndices = new ArrayListInteger();
private final ListInteger byteFieldIndices = new ArrayListInteger();
- private Schema fromSchema;
- private Schema toSchema;
+ private Schema schema;
/**
* {@inheritDoc}
@@ -94,11 +87,11 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
* {@inheritDoc}
*/
@Override
- public void setFromSchema(Schema schema) {
+ public void setSchema(Schema schema) {
if(schema == null) {
return;
}
-this.fromSchema = schema;
+this.schema = schema;
ListColumn columns = schema.getColumns();
int i = 0;
for(Column col : columns) {
@@ -112,19 +105,6 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
}
/**
- * {@inheritDoc}
- */
- @Override
- public void setToSchema(Schema schema) {
-if(schema == null) {
- return;
-}
-this.toSchema = schema;
- }
-
-
-
- /**
* Custom CSV parser that honors quoting and escaped quotes.
* All other escaping is handled elsewhere.
*
@@ -180,69 +160,68 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
/**
* {@inheritDoc}
- *
- * The CSV data is ordered according to the fromSchema. We translate it to
the TO schema.
- * We currently have 3 methods of matching fields in one schema to another:
- * - by location
- * - by name
- * - user-defined matching
- *
- * If one
[23/52] [abbrv] git commit: SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/3d539dd4
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/3d539dd4
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/3d539dd4
Branch: refs/heads/SQOOP-1367
Commit: 3d539dd4d7477324dfe62a4e57f684351769b000
Parents: af25bcc
Author: Abraham Elmahrek abra...@elmahrek.com
Authored: Fri Sep 19 16:24:59 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../org/apache/sqoop/json/SubmissionBean.java | 20 +-
.../org/apache/sqoop/model/MSubmission.java | 31 +-
.../apache/sqoop/json/TestSubmissionBean.java | 12 +-
.../sqoop/connector/jdbc/TestToInitializer.java | 1 -
.../idf/CSVIntermediateDataFormat.java | 4 -
.../connector/idf/IntermediateDataFormat.java | 4 -
.../idf/CSVIntermediateDataFormatTest.java | 222 --
.../idf/TestCSVIntermediateDataFormat.java | 222 ++
.../sqoop/connector/ConnectorManager.java | 5 +-
.../apache/sqoop/framework/ExecutionEngine.java | 20 +-
.../org/apache/sqoop/framework/JobManager.java | 447 ++--
.../org/apache/sqoop/framework/JobRequest.java | 356 ++
.../sqoop/framework/SubmissionEngine.java | 7 +-
.../sqoop/framework/SubmissionRequest.java | 361 --
.../sqoop/framework/TestFrameworkValidator.java | 182 +++--
.../apache/sqoop/framework/TestJobManager.java | 173 +
.../apache/sqoop/framework/TestJobRequest.java | 71 ++
.../sqoop/framework/TestSubmissionRequest.java | 71 --
.../sqoop/repository/TestJdbcRepository.java| 694 +--
.../sqoop/execution/mapreduce/MRJobRequest.java | 102 +++
.../mapreduce/MRSubmissionRequest.java | 102 ---
.../mapreduce/MapreduceExecutionEngine.java | 51 +-
.../apache/sqoop/job/mr/ConfigurationUtils.java | 3 -
.../apache/sqoop/job/mr/ProgressRunnable.java | 4 +-
.../sqoop/job/mr/SqoopDestroyerExecutor.java| 7 +-
.../sqoop/job/mr/SqoopFileOutputFormat.java | 4 +-
.../org/apache/sqoop/job/mr/SqoopMapper.java| 50 +-
.../job/mr/SqoopOutputFormatLoadExecutor.java | 28 +-
.../sqoop/shell/utils/SubmissionDisplayer.java | 8 +-
.../org/apache/sqoop/job/etl/CallbackBase.java | 49 --
.../java/org/apache/sqoop/job/etl/From.java | 2 +-
.../main/java/org/apache/sqoop/job/etl/To.java | 2 +-
.../org/apache/sqoop/job/etl/Transferable.java | 51 ++
.../org/apache/sqoop/validation/Validator.java | 1 -
.../mapreduce/MapreduceSubmissionEngine.java| 12 +-
35 files changed, 1780 insertions(+), 1599 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3d539dd4/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
--
diff --git a/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
b/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
index 61d6576..9b1ae74 100644
--- a/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
+++ b/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
@@ -52,8 +52,8 @@ public class SubmissionBean implements JsonBean {
private static final String EXCEPTION_TRACE = exception-trace;
private static final String PROGRESS = progress;
private static final String COUNTERS = counters;
- private static final String CONNECTOR_SCHEMA = schema-connector;
- private static final String HIO_SCHEMA = schema-hio;
+ private static final String FROM_SCHEMA = schema-from;
+ private static final String TO_SCHEMA = schema-to;
private ListMSubmission submissions;
@@ -116,11 +116,11 @@ public class SubmissionBean implements JsonBean {
if(submission.getCounters() != null) {
object.put(COUNTERS, extractCounters(submission.getCounters()));
}
- if(submission.getConnectorSchema() != null) {
-object.put(CONNECTOR_SCHEMA,
extractSchema(submission.getConnectorSchema()));
+ if(submission.getFromSchema() != null) {
+object.put(FROM_SCHEMA, extractSchema(submission.getFromSchema()));
}
- if(submission.getHioSchema() != null) {
-object.put(HIO_SCHEMA, extractSchema(submission.getHioSchema()));
+ if(submission.getToSchema() != null) {
+object.put(TO_SCHEMA, extractSchema(submission.getToSchema()));
}
array.add(object);
@@ -188,11 +188,11 @@ public class SubmissionBean implements JsonBean {
if(object.containsKey(COUNTERS)) {
submission.setCounters(restoreCounters((JSONObject)
object.get(COUNTERS)));
}
- if(object.containsKey(CONNECTOR_SCHEMA)) {
-
[11/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/core/src/main/java/org/apache/sqoop/repository/Repository.java
--
diff --git a/core/src/main/java/org/apache/sqoop/repository/Repository.java
b/core/src/main/java/org/apache/sqoop/repository/Repository.java
index 8e8dd80..ea1608a 100644
--- a/core/src/main/java/org/apache/sqoop/repository/Repository.java
+++ b/core/src/main/java/org/apache/sqoop/repository/Repository.java
@@ -17,36 +17,36 @@
*/
package org.apache.sqoop.repository;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
import org.apache.log4j.Logger;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.connector.spi.MetadataUpgrader;
+import org.apache.sqoop.connector.spi.RepositoryUpgrader;
import org.apache.sqoop.connector.spi.SqoopConnector;
-import org.apache.sqoop.framework.FrameworkManager;
+import org.apache.sqoop.driver.Driver;
import org.apache.sqoop.model.FormUtils;
-import org.apache.sqoop.model.MConnection;
import org.apache.sqoop.model.MConnectionForms;
import org.apache.sqoop.model.MConnector;
+import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MFramework;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MJobForms;
+import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MPersistableEntity;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.utils.ClassUtils;
import org.apache.sqoop.validation.Validation;
import org.apache.sqoop.validation.Validator;
-import java.util.Date;
-import java.util.List;
-import java.util.Map;
-
/**
- * Defines the contract of a Repository used by Sqoop. A Repository allows
- * Sqoop to store metadata, statistics and other state relevant to Sqoop
- * Jobs in the system.
+ * Defines the contract for repository used by Sqoop. A Repository allows
+ * Sqoop to store entities such as connectors, links, jobs, submissions and
its related configs,
+ * statistics and other state relevant the entities in the store
*/
public abstract class Repository {
@@ -76,11 +76,11 @@ public abstract class Repository {
/**
* Registers given connector in the repository and return registered
- * variant. This method might return an exception in case that metadata for
+ * variant. This method might return an exception in case that
* given connector are already registered with different structure.
*
- * @param mConnector the connector metadata to be registered
- * autoupgrade whether to upgrade framework automatically
+ * @param mConnector the connector to be registered
+ * autoupgrade whether to upgrade driver config automatically
* @return Registered connector structure
*/
public abstract MConnector registerConnector(MConnector mConnector, boolean
autoUpgrade);
@@ -88,7 +88,7 @@ public abstract class Repository {
/**
* Search for connector with given name in repository.
*
- * And return corresponding metadata structure.
+ * And return corresponding entity structure.
*
* @param shortName Connector unique name
* @return null if connector is not yet registered in repository or
@@ -105,76 +105,75 @@ public abstract class Repository {
/**
- * Registers given framework in the repository and return registered
- * variant. This method might return an exception in case that metadata for
- * given framework are already registered with different structure.
+ * Registers given driverConfig in the repository and return registered
+ * variant. This method might return an exception in case that the
+ * given driverConfig are already registered with different structure.
*
- * @param mFramework framework metadata to be registered
- * autoupgrade whether to upgrade framework automatically
+ * @param mDriverConfig driverConfig to be registered
+ * autoupgrade whether to upgrade driverConfig automatically
* @return Registered connector structure
*/
- public abstract MFramework registerFramework(MFramework mFramework, boolean
autoUpgrade);
+ public abstract MDriverConfig registerDriverConfig(MDriverConfig
mDriverConfig, boolean autoUpgrade);
/**
- * Save given connection to repository. This connection must not be already
+ * Save given link to repository. This link must not be already
* present in the repository otherwise exception will be thrown.
*
- * @param connection Connection object to serialize into repository.
+ * @param link link object to serialize into repository.
*/
- public abstract void createConnection(MConnection connection);
+ public abstract void createLink(MLink link);
/**
- * Update given connection representation in repository. This connection
+ * Update given link representation in
[07/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
--
diff --git
a/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
deleted file mode 100644
index 1eb7e51..000
--- a/shell/src/main/java/org/apache/sqoop/shell/DeleteConnectionFunction.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.shell;
-
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.cli.OptionBuilder;
-import org.apache.sqoop.shell.core.Constants;
-import org.apache.sqoop.validation.Status;
-
-import static org.apache.sqoop.shell.ShellEnvironment.*;
-
-/**
- *
- */
-@SuppressWarnings(serial)
-public class DeleteConnectionFunction extends SqoopFunction {
- @SuppressWarnings(static-access)
- public DeleteConnectionFunction() {
-this.addOption(OptionBuilder
- .withDescription(resourceString(Constants.RES_PROMPT_CONN_ID))
- .withLongOpt(Constants.OPT_XID)
- .isRequired()
- .hasArg()
- .create('x'));
- }
-
- @Override
- public Object executeFunction(CommandLine line, boolean isInteractive) {
-client.deleteConnection(getLong(line, Constants.OPT_XID));
-return Status.FINE;
- }
-}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
index 5d48c91..da26153 100644
--- a/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
+++ b/shell/src/main/java/org/apache/sqoop/shell/DeleteJobFunction.java
@@ -37,7 +37,7 @@ public class DeleteJobFunction extends SqoopFunction {
.withLongOpt(Constants.OPT_JID)
.isRequired()
.hasArg()
- .create('j'));
+ .create(Constants.OPT_JID_CHAR));
}
@Override
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
b/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
new file mode 100644
index 000..c81917e
--- /dev/null
+++ b/shell/src/main/java/org/apache/sqoop/shell/DeleteLinkFunction.java
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.shell;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.OptionBuilder;
+import org.apache.sqoop.shell.core.Constants;
+import org.apache.sqoop.validation.Status;
+
+import static org.apache.sqoop.shell.ShellEnvironment.*;
+
+/**
+ *
+ */
+@SuppressWarnings(serial)
+public class DeleteLinkFunction extends SqoopFunction {
+ @SuppressWarnings(static-access)
+ public DeleteLinkFunction() {
+this.addOption(OptionBuilder
+ .withDescription(resourceString(Constants.RES_PROMPT_LINK_ID))
+ .withLongOpt(Constants.OPT_LID)
+ .isRequired()
+ .hasArg()
+ .create(Constants.OPT_LID_CHAR));
+ }
+
+ @Override
+ public Object executeFunction(CommandLine
[10/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
--
diff --git
a/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
b/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
index 0fcae75..796b021 100644
--- a/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
+++ b/core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
@@ -17,6 +17,23 @@
*/
package org.apache.sqoop.repository;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Matchers.any;
+import static org.mockito.Matchers.anyLong;
+import static org.mockito.Matchers.anyObject;
+import static org.mockito.Matchers.anyString;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.doReturn;
+import static org.mockito.Mockito.doThrow;
+import static org.mockito.Mockito.inOrder;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
import java.sql.Connection;
import java.util.ArrayList;
import java.util.Collections;
@@ -26,39 +43,35 @@ import java.util.List;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.connector.spi.MetadataUpgrader;
+import org.apache.sqoop.connector.spi.RepositoryUpgrader;
import org.apache.sqoop.connector.spi.SqoopConnector;
-import org.apache.sqoop.framework.FrameworkManager;
-import org.apache.sqoop.framework.configuration.JobConfiguration;
+import org.apache.sqoop.driver.Driver;
+import org.apache.sqoop.driver.configuration.JobConfiguration;
import org.apache.sqoop.model.ConfigurationClass;
import org.apache.sqoop.model.FormUtils;
-import org.apache.sqoop.model.MConnection;
import org.apache.sqoop.model.MConnectionForms;
import org.apache.sqoop.model.MConnector;
+import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MFramework;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MJobForms;
+import org.apache.sqoop.model.MLink;
import org.apache.sqoop.validation.Status;
import org.apache.sqoop.validation.Validation;
-
import org.apache.sqoop.validation.Validator;
import org.junit.Before;
import org.junit.Test;
import org.mockito.InOrder;
-import static org.junit.Assert.*;
-import static org.mockito.Mockito.*;
-
public class TestJdbcRepository {
private JdbcRepository repoSpy;
private JdbcRepositoryTransaction repoTransactionMock;
private ConnectorManager connectorMgrMock;
- private FrameworkManager frameworkMgrMock;
+ private Driver driverMock;
private JdbcRepositoryHandler repoHandlerMock;
private Validator validatorMock;
- private MetadataUpgrader upgraderMock;
+ private RepositoryUpgrader upgraderMock;
private Validation validRepoMock;
private Validation invalidRepoMock;
@@ -67,16 +80,16 @@ public class TestJdbcRepository {
public void setUp() throws Exception {
repoTransactionMock = mock(JdbcRepositoryTransaction.class);
connectorMgrMock = mock(ConnectorManager.class);
-frameworkMgrMock = mock(FrameworkManager.class);
+driverMock = mock(Driver.class);
repoHandlerMock = mock(JdbcRepositoryHandler.class);
validatorMock = mock(Validator.class);
-upgraderMock = mock(MetadataUpgrader.class);
+upgraderMock = mock(RepositoryUpgrader.class);
repoSpy = spy(new JdbcRepository(repoHandlerMock, null));
// setup transaction and connector manager
doReturn(repoTransactionMock).when(repoSpy).getTransaction();
ConnectorManager.setInstance(connectorMgrMock);
-FrameworkManager.setInstance(frameworkMgrMock);
+Driver.setInstance(driverMock);
validRepoMock = mock(Validation.class);
when(validRepoMock.getStatus()).thenReturn(Status.ACCEPTABLE);
@@ -138,26 +151,26 @@ public class TestJdbcRepository {
}
/**
- * Test the procedure when the framework auto upgrade option is enabled
+ * Test the procedure when the driverConfig auto upgrade option is enabled
*/
@Test
- public void testFrameworkEnableAutoUpgrade() {
-MFramework newFramework = framework();
-MFramework oldFramework = anotherFramework();
+ public void testDriverConfigEnableAutoUpgrade() {
+MDriverConfig newDriverConfig = driverConfig();
+MDriverConfig oldDriverConfig = anotherDriverConfig();
-
when(repoHandlerMock.findFramework(any(Connection.class))).thenReturn(oldFramework);
+
when(repoHandlerMock.findDriverConfig(any(Connection.class))).thenReturn(oldDriverConfig);
-// make the upgradeFramework to throw
[08/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
new file mode 100644
index 000..7378eda
--- /dev/null
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
@@ -0,0 +1,244 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.repository.derby;
+
+import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.model.MLink;
+import org.apache.sqoop.model.MForm;
+import org.apache.sqoop.model.MMapInput;
+import org.apache.sqoop.model.MStringInput;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Test link methods on Derby repository.
+ */
+public class TestLinkHandling extends DerbyTestCase {
+
+ DerbyRepositoryHandler handler;
+
+ @Override
+ public void setUp() throws Exception {
+super.setUp();
+
+handler = new DerbyRepositoryHandler();
+
+// We always needs schema for this test case
+createSchema();
+
+// We always needs connector and framework structures in place
+loadConnectorAndDriverConfig();
+ }
+
+ public void testFindLink() throws Exception {
+// Let's try to find non existing link
+try {
+ handler.findLink(1, getDerbyDatabaseConnection());
+ fail();
+} catch(SqoopException ex) {
+ assertEquals(DerbyRepoError.DERBYREPO_0024, ex.getErrorCode());
+}
+
+// Load prepared connections into database
+loadLinks();
+
+MLink connA = handler.findLink(1, getDerbyDatabaseConnection());
+assertNotNull(connA);
+assertEquals(1, connA.getPersistenceId());
+assertEquals(CA, connA.getName());
+
+ListMForm forms;
+
+// Check connector part
+forms = connA.getConnectorPart().getForms();
+assertEquals(Value1, forms.get(0).getInputs().get(0).getValue());
+assertNull(forms.get(0).getInputs().get(1).getValue());
+assertEquals(Value3, forms.get(1).getInputs().get(0).getValue());
+assertNull(forms.get(1).getInputs().get(1).getValue());
+
+// Check framework part
+forms = connA.getFrameworkPart().getForms();
+assertEquals(Value13, forms.get(0).getInputs().get(0).getValue());
+assertNull(forms.get(0).getInputs().get(1).getValue());
+assertEquals(Value15, forms.get(1).getInputs().get(0).getValue());
+assertNull(forms.get(1).getInputs().get(1).getValue());
+ }
+
+ public void testFindLinks() throws Exception {
+ListMLink list;
+
+// Load empty list on empty repository
+list = handler.findLinks(getDerbyDatabaseConnection());
+assertEquals(0, list.size());
+
+loadLinks();
+
+// Load all two connections on loaded repository
+list = handler.findLinks(getDerbyDatabaseConnection());
+assertEquals(2, list.size());
+
+assertEquals(CA, list.get(0).getName());
+assertEquals(CB, list.get(1).getName());
+ }
+
+ public void testExistsLink() throws Exception {
+// There shouldn't be anything on empty repository
+assertFalse(handler.existsLink(1, getDerbyDatabaseConnection()));
+assertFalse(handler.existsLink(2, getDerbyDatabaseConnection()));
+assertFalse(handler.existsLink(3, getDerbyDatabaseConnection()));
+
+loadLinks();
+
+assertTrue(handler.existsLink(1, getDerbyDatabaseConnection()));
+assertTrue(handler.existsLink(2, getDerbyDatabaseConnection()));
+assertFalse(handler.existsLink(3, getDerbyDatabaseConnection()));
+ }
+
+ public void testCreateLink() throws Exception {
+MLink link = getLink();
+
+// Load some data
+fillLink(link);
+
+handler.createLink(link, getDerbyDatabaseConnection());
+
+assertEquals(1, link.getPersistenceId());
+assertCountForTable(SQOOP.SQ_CONNECTION, 1);
+assertCountForTable(SQOOP.SQ_CONNECTION_INPUT, 4);
+
+MLink retrieved = handler.findLink(1, getDerbyDatabaseConnection());
+assertEquals(1,
[22/50] [abbrv] git commit: SQOOP-1454: Sqoop2: From/To: Add client support for supported directions
SQOOP-1454: Sqoop2: From/To: Add client support for supported directions
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/27fb31d4
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/27fb31d4
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/27fb31d4
Branch: refs/heads/sqoop2
Commit: 27fb31d42e7e76ca14def568e454121437f4e9e5
Parents: 3d539dd
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Sun Sep 21 12:42:10 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../sqoop/common/SupportedDirections.java | 41 ++
.../java/org/apache/sqoop/model/MConnector.java | 6 +
.../sqoop/common/TestSupportedDirections.java | 55
.../org/apache/sqoop/model/TestMConnector.java | 127 ---
.../sqoop/connector/ConnectorHandler.java | 4 -
.../sqoop/shell/ShowConnectorFunction.java | 37 +-
.../org/apache/sqoop/shell/core/Constants.java | 2 +
.../main/resources/shell-resource.properties| 5 +-
8 files changed, 226 insertions(+), 51 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
new file mode 100644
index 000..25ba276
--- /dev/null
+++ b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
@@ -0,0 +1,41 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.common;
+
+/**
+ * Represents which Directions are supported.
+ */
+public class SupportedDirections {
+ private boolean from;
+ private boolean to;
+
+ public SupportedDirections(boolean from, boolean to) {
+this.from = from;
+this.to = to;
+ }
+
+ /**
+ * Check if direction is supported.
+ * @param direction
+ * @return boolean
+ */
+ public boolean isDirectionSupported(Direction direction) {
+return direction == Direction.FROM from
+|| direction == Direction.TO to;
+ }
+}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/main/java/org/apache/sqoop/model/MConnector.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnector.java
b/common/src/main/java/org/apache/sqoop/model/MConnector.java
index 335a0cc..3dc1014 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnector.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnector.java
@@ -20,6 +20,7 @@ package org.apache.sqoop.model;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.DirectionError;
import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.common.SupportedDirections;
/**
* Connector metadata.
@@ -139,4 +140,9 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
public void setVersion(String version) {
this.version = version;
}
+
+ public SupportedDirections getSupportedDirections() {
+return new SupportedDirections(this.getJobForms(Direction.FROM) != null,
+this.getJobForms(Direction.TO) != null);
+ }
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/27fb31d4/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
--
diff --git
a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
new file mode 100644
index 000..4fbaf82
--- /dev/null
+++ b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
@@ -0,0 +1,55 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE
[20/50] [abbrv] SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3d539dd4/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
--
diff --git a/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
b/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
new file mode 100644
index 000..2732b1c
--- /dev/null
+++ b/core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
@@ -0,0 +1,173 @@
+package org.apache.sqoop.framework;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.fail;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.times;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.sqoop.common.Direction;
+import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.connector.ConnectorManager;
+import org.apache.sqoop.connector.spi.SqoopConnector;
+import org.apache.sqoop.model.MConnection;
+import org.apache.sqoop.model.MJob;
+import org.apache.sqoop.model.MSubmission;
+import org.apache.sqoop.repository.JdbcRepository;
+import org.apache.sqoop.repository.Repository;
+import org.apache.sqoop.repository.RepositoryManager;
+import org.apache.sqoop.request.HttpEventContext;
+import org.junit.Before;
+import org.junit.Test;
+
+public class TestJobManager {
+ private JobManager jobManager;
+ private SqoopConnector sqoopConnectorMock;
+ private ConnectorManager connectorMgrMock;
+ private RepositoryManager repositoryManagerMock;
+ private Repository jdbcRepoMock;
+
+ @Before
+ public void setUp() {
+jobManager = new JobManager();
+connectorMgrMock = mock(ConnectorManager.class);
+sqoopConnectorMock = mock(SqoopConnector.class);
+ConnectorManager.setInstance(connectorMgrMock);
+repositoryManagerMock = mock(RepositoryManager.class);
+RepositoryManager.setInstance(repositoryManagerMock);
+jdbcRepoMock = mock(JdbcRepository.class);
+ }
+
+ @Test
+ public void testCreateJobSubmission() {
+
+HttpEventContext testCtx = new HttpEventContext();
+testCtx.setUsername(testUser);
+MSubmission jobSubmission = jobManager.createJobSubmission(testCtx, 1234L);
+assertEquals(jobSubmission.getCreationUser(), testUser);
+assertEquals(jobSubmission.getLastUpdateUser(), testUser);
+ }
+
+ @Test
+ public void testGetConnector() {
+when(connectorMgrMock.getConnector(123l)).thenReturn(sqoopConnectorMock);
+
when(sqoopConnectorMock.getSupportedDirections()).thenReturn(getSupportedDirections());
+assertEquals(jobManager.getConnector(123l), sqoopConnectorMock);
+verify(connectorMgrMock, times(1)).getConnector(123l);
+ }
+
+ @Test
+ public void testUnsupportedDirectionForConnector() {
+// invalid job id/ direction
+SqoopException exception = new
SqoopException(FrameworkError.FRAMEWORK_0011, Connector:
++ sqoopConnectorMock.getClass().getCanonicalName());
+ListDirection supportedDirections = getSupportedDirections();
+
when(sqoopConnectorMock.getSupportedDirections()).thenReturn(supportedDirections);
+
+try {
+ // invalid direction
+ jobManager.validateSupportedDirection(sqoopConnectorMock, null);
+} catch (SqoopException ex) {
+ assertEquals(ex.getMessage(), exception.getMessage());
+ verify(sqoopConnectorMock, times(1)).getSupportedDirections();
+ return;
+}
+
+fail(Should throw out an exception with message: +
exception.getMessage());
+ }
+
+ @Test
+ public void testGetConnection() {
+MConnection testConnection = new MConnection(123l, null, null);
+testConnection.setEnabled(true);
+MConnection mConnectionSpy = org.mockito.Mockito.spy(testConnection);
+when(repositoryManagerMock.getRepository()).thenReturn(jdbcRepoMock);
+when(jdbcRepoMock.findConnection(123l)).thenReturn(mConnectionSpy);
+assertEquals(jobManager.getConnection(123l), mConnectionSpy);
+verify(repositoryManagerMock, times(1)).getRepository();
+verify(jdbcRepoMock, times(1)).findConnection(123l);
+ }
+
+ @Test
+ public void testDisabledConnection() {
+MConnection testConnection = new MConnection(123l, null, null);
+testConnection.setPersistenceId(1234);
+testConnection.setEnabled(false);
+SqoopException exception = new
SqoopException(FrameworkError.FRAMEWORK_0010, Connection id:
++ testConnection.getPersistenceId());
+
+MConnection mConnectionSpy = org.mockito.Mockito.spy(testConnection);
+when(repositoryManagerMock.getRepository()).thenReturn(jdbcRepoMock);
+when(jdbcRepoMock.findConnection(123l)).thenReturn(mConnectionSpy);
+try {
+ jobManager.getConnection(123l);
+} catch (SqoopException ex) {
+ assertEquals(ex.getMessage(), exception.getMessage());
+ verify(repositoryManagerMock, times(1)).getRepository();
+ verify(jdbcRepoMock,
[12/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/core/src/main/java/org/apache/sqoop/framework/JobManager.java
--
diff --git a/core/src/main/java/org/apache/sqoop/framework/JobManager.java
b/core/src/main/java/org/apache/sqoop/framework/JobManager.java
deleted file mode 100644
index 8149d1c..000
--- a/core/src/main/java/org/apache/sqoop/framework/JobManager.java
+++ /dev/null
@@ -1,710 +0,0 @@
-
-
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.framework;
-
-import org.apache.log4j.Logger;
-import org.apache.sqoop.common.Direction;
-import org.apache.sqoop.common.MapContext;
-import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.framework.configuration.JobConfiguration;
-import org.apache.sqoop.request.HttpEventContext;
-import org.apache.sqoop.connector.idf.IntermediateDataFormat;
-import org.apache.sqoop.connector.spi.SqoopConnector;
-import org.apache.sqoop.core.Reconfigurable;
-import org.apache.sqoop.core.SqoopConfiguration;
-import org.apache.sqoop.core.SqoopConfiguration.CoreConfigurationListener;
-import org.apache.sqoop.job.etl.*;
-import org.apache.sqoop.model.FormUtils;
-import org.apache.sqoop.model.MConnection;
-import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MSubmission;
-import org.apache.sqoop.repository.Repository;
-import org.apache.sqoop.repository.RepositoryManager;
-import org.apache.sqoop.schema.Schema;
-import org.apache.sqoop.submission.SubmissionStatus;
-import org.apache.sqoop.submission.counter.Counters;
-import org.apache.sqoop.utils.ClassUtils;
-import org.json.simple.JSONValue;
-
-import java.util.Date;
-import java.util.List;
-
-public class JobManager implements Reconfigurable {
- /**
- * Logger object.
- */
- private static final Logger LOG = Logger.getLogger(JobManager.class);
-
- /**
- * Private instance to singleton of this class.
- */
- private static JobManager instance;
- /**
- * Create default object by default.
- *
- * Every Sqoop server application needs one so this should not be performance
- * issue.
- */
- static {
-instance = new JobManager();
- }
-
- /**
- * Return current instance.
- *
- * @return Current instance
- */
- public static JobManager getInstance() {
-return instance;
- }
-
- /**
- * Allows to set instance in case that it's need.
- *
- * This method should not be normally used as the default instance should be
- * sufficient. One target user use case for this method are unit tests.
- *
- * @param newInstance
- * New instance
- */
- public static void setInstance(JobManager newInstance) {
-instance = newInstance;
- }
-
- /**
- * Default interval for purging old submissions from repository.
- */
- private static final long DEFAULT_PURGE_THRESHOLD = 24 * 60 * 60 * 1000;
-
- /**
- * Default sleep interval for purge thread.
- */
- private static final long DEFAULT_PURGE_SLEEP = 24 * 60 * 60 * 1000;
-
- /**
- * Default interval for update thread.
- */
- private static final long DEFAULT_UPDATE_SLEEP = 60 * 5 * 1000;
-
- /**
- * Configured submission engine instance
- */
- private SubmissionEngine submissionEngine;
-
- /**
- * Configured execution engine instance
- */
- private ExecutionEngine executionEngine;
-
- /**
- * Purge thread that will periodically remove old submissions from
repository.
- */
- private PurgeThread purgeThread = null;
-
- /**
- * Update thread that will periodically check status of running submissions.
- */
- private UpdateThread updateThread = null;
-
- /**
- * Synchronization variable between threads.
- */
- private boolean running = true;
-
- /**
- * Specifies how old submissions should be removed from repository.
- */
- private long purgeThreshold;
-
- /**
- * Number of milliseconds for purge thread to sleep.
- */
- private long purgeSleep;
-
- /**
- * Number of milliseconds for update thread to slepp.
- */
- private long updateSleep;
-
- /**
- * Base notification URL.
- *
- * Framework manager will always add job id.
- */
-
[01/50] [abbrv] git commit: SQOOP-1467: Sqoop2: Make MAccountableEntity inherit from MValidatedElement rather then MPersistableEntity
Repository: sqoop
Updated Branches:
refs/heads/sqoop2 6a9c93d6e - d7e049043
SQOOP-1467: Sqoop2: Make MAccountableEntity inherit from MValidatedElement
rather then MPersistableEntity
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/58bb8dd0
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/58bb8dd0
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/58bb8dd0
Branch: refs/heads/sqoop2
Commit: 58bb8dd07a0288d30d20755df5ee69f7990a7061
Parents: 2873a4b
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Mon Aug 25 11:04:43 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
.../apache/sqoop/model/MAccountableEntity.java | 3 ++-
.../org/apache/sqoop/model/MConnection.java | 12 +-
.../main/java/org/apache/sqoop/model/MJob.java | 15
.../org/apache/sqoop/model/MNamedElement.java | 25 +++-
.../apache/sqoop/model/MValidatedElement.java | 6 +
5 files changed, 28 insertions(+), 33 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/58bb8dd0/common/src/main/java/org/apache/sqoop/model/MAccountableEntity.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/model/MAccountableEntity.java
b/common/src/main/java/org/apache/sqoop/model/MAccountableEntity.java
index 781331b..094af39 100644
--- a/common/src/main/java/org/apache/sqoop/model/MAccountableEntity.java
+++ b/common/src/main/java/org/apache/sqoop/model/MAccountableEntity.java
@@ -23,7 +23,7 @@ import java.util.Date;
* Accountable entity provides additional fields that might help with
identifying
* what and when has happened.
*/
-abstract public class MAccountableEntity extends MPersistableEntity {
+abstract public class MAccountableEntity extends MValidatedElement {
private final boolean DEFAULT_ENABLED = true;
@@ -59,6 +59,7 @@ abstract public class MAccountableEntity extends
MPersistableEntity {
* the accountable entity is enabled.
*/
public MAccountableEntity() {
+super((String)null);
this.creationUser = null;
this.creationDate = new Date();
this.lastUpdateUser = this.creationUser;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/58bb8dd0/common/src/main/java/org/apache/sqoop/model/MConnection.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnection.java
b/common/src/main/java/org/apache/sqoop/model/MConnection.java
index da9691f..e5a4fb8 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnection.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnection.java
@@ -23,7 +23,6 @@ package org.apache.sqoop.model;
*/
public class MConnection extends MAccountableEntity implements MClonable {
private long connectorId;
- private String name;
private final MConnectionForms connectorPart;
private final MConnectionForms frameworkPart;
@@ -65,28 +64,19 @@ public class MConnection extends MAccountableEntity
implements MClonable {
public MConnection(MConnection other, MConnectionForms connectorPart,
MConnectionForms frameworkPart) {
super(other);
this.connectorId = other.connectorId;
-this.name = other.name;
this.connectorPart = connectorPart;
this.frameworkPart = frameworkPart;
}
@Override
public String toString() {
-StringBuilder sb = new StringBuilder(connection: ).append(name);
+StringBuilder sb = new StringBuilder(connection: ).append(getName());
sb.append( connector-part: ).append(connectorPart);
sb.append(, framework-part: ).append(frameworkPart);
return sb.toString();
}
- public String getName() {
-return name;
- }
-
- public void setName(String name) {
-this.name = name;
- }
-
public long getConnectorId() {
return connectorId;
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/58bb8dd0/common/src/main/java/org/apache/sqoop/model/MJob.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MJob.java
b/common/src/main/java/org/apache/sqoop/model/MJob.java
index 3835113..11839fc 100644
--- a/common/src/main/java/org/apache/sqoop/model/MJob.java
+++ b/common/src/main/java/org/apache/sqoop/model/MJob.java
@@ -42,11 +42,6 @@ public class MJob extends MAccountableEntity implements
MClonable {
private final long fromConnectionId;
private final long toConnectionId;
- /**
- * User name for this object
- */
- private String name;
-
private final MJobForms fromConnectorPart;
private final MJobForms toConnectorPart;
private final MJobForms frameworkPart;
@@ -104,8 +99,6 @@ public class MJob extends
[09/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
index 68aea9c..5dd7970 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
@@ -30,8 +30,6 @@ import java.sql.Timestamp;
import java.sql.Types;
import java.util.*;
-import javax.sql.DataSource;
-
import org.apache.log4j.Logger;
import org.apache.commons.lang.StringUtils;
import org.apache.sqoop.common.Direction;
@@ -40,7 +38,7 @@ import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorHandler;
import org.apache.sqoop.connector.ConnectorManagerUtils;
import org.apache.sqoop.model.MBooleanInput;
-import org.apache.sqoop.model.MConnection;
+import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MConnectionForms;
import org.apache.sqoop.model.MEnumInput;
import org.apache.sqoop.model.MIntegerInput;
@@ -49,7 +47,7 @@ import org.apache.sqoop.model.MJobForms;
import org.apache.sqoop.model.MConnector;
import org.apache.sqoop.model.MForm;
import org.apache.sqoop.model.MFormType;
-import org.apache.sqoop.model.MFramework;
+import org.apache.sqoop.model.MDriverConfig;
import org.apache.sqoop.model.MInput;
import org.apache.sqoop.model.MInputType;
import org.apache.sqoop.model.MMapInput;
@@ -57,7 +55,6 @@ import org.apache.sqoop.model.MStringInput;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.repository.JdbcRepositoryContext;
import org.apache.sqoop.repository.JdbcRepositoryHandler;
-import org.apache.sqoop.repository.JdbcRepositoryTransactionFactory;
import org.apache.sqoop.submission.SubmissionStatus;
import org.apache.sqoop.submission.counter.Counter;
import org.apache.sqoop.submission.counter.CounterGroup;
@@ -78,15 +75,13 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
/**
* Unique name of HDFS Connector.
- * HDFS Connector was originally part of the Sqoop framework, but now is its
+ * HDFS Connector was originally part of the Sqoop driver, but now is its
* own connector. This constant is used to pre-register the HDFS Connector
* so that jobs that are being upgraded can reference the HDFS Connector.
*/
private static final String CONNECTOR_HDFS = hdfs-connector;
private JdbcRepositoryContext repoContext;
- private DataSource dataSource;
- private JdbcRepositoryTransactionFactory txFactory;
/**
* {@inheritDoc}
@@ -105,10 +100,10 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
* Helper method to insert the forms from the into the
* repository. The job and connector forms within codemc/code will get
* updated with the id of the forms when this function returns.
- * @param mf The MFramework instance to use to upgrade.
- * @param conn JDBC connection to use for updating the forms
+ * @param mDriverConfig The MFramework instance to use to upgrade.
+ * @param conn JDBC link to use for updating the forms
*/
- private void insertFormsForFramework(MFramework mf, Connection conn) {
+ private void insertFormsForFramework(MDriverConfig mDriverConfig, Connection
conn) {
PreparedStatement baseFormStmt = null;
PreparedStatement baseInputStmt = null;
try{
@@ -119,15 +114,15 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
Statement.RETURN_GENERATED_KEYS);
// Register connector forms
- registerForms(null, null, mf.getConnectionForms().getForms(),
+ registerForms(null, null, mDriverConfig.getConnectionForms().getForms(),
MFormType.CONNECTION.name(), baseFormStmt, baseInputStmt);
// Register job forms
- registerForms(null, null, mf.getJobForms().getForms(),
+ registerForms(null, null, mDriverConfig.getJobForms().getForms(),
MFormType.JOB.name(), baseFormStmt, baseInputStmt);
} catch (SQLException ex) {
- throw new SqoopException(DerbyRepoError.DERBYREPO_0014, mf.toString(),
ex);
+ throw new SqoopException(DerbyRepoError.DERBYREPO_0014,
mDriverConfig.toString(), ex);
} finally {
closeStatements(baseFormStmt, baseInputStmt);
}
@@ -138,7 +133,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
* repository. The job and connector forms within codemc/code will get
* updated with the id of the forms when this function returns.
* @param mc The connector to use for updating forms
- * @param conn JDBC connection to use for updating the
[02/50] [abbrv] git commit: SQOOP-1487: Sqoop2: From/To: Refactor/Create HDFS connector test cases
SQOOP-1487: Sqoop2: From/To: Refactor/Create HDFS connector test cases
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/c0b22b1d
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/c0b22b1d
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/c0b22b1d
Branch: refs/heads/sqoop2
Commit: c0b22b1d62221f2a24294520b8ff3851c06d06c3
Parents: 8fee134
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Wed Sep 3 09:40:11 2014 +0200
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
connector/connector-hdfs/pom.xml| 5 +
.../sqoop/connector/hdfs/HdfsExtractor.java | 2 -
.../apache/sqoop/connector/hdfs/FileUtils.java | 82 ++
.../sqoop/connector/hdfs/TestExtractor.java | 125 ++
.../sqoop/connector/hdfs/TestHdfsBase.java | 139 +++
.../apache/sqoop/connector/hdfs/TestLoader.java | 213
.../sqoop/connector/hdfs/TestPartitioner.java | 113 +
.../src/test/resources/log4j.properties | 24 ++
.../java/org/apache/sqoop/job/FileUtils.java| 69 -
.../org/apache/sqoop/job/TestHdfsExtract.java | 241 --
.../java/org/apache/sqoop/job/TestHdfsLoad.java | 250 ---
11 files changed, 701 insertions(+), 562 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/pom.xml
--
diff --git a/connector/connector-hdfs/pom.xml b/connector/connector-hdfs/pom.xml
index 8df9f11..fa4330a 100644
--- a/connector/connector-hdfs/pom.xml
+++ b/connector/connector-hdfs/pom.xml
@@ -35,6 +35,11 @@ limitations under the License.
dependencies
dependency
+ groupIdjunit/groupId
+ artifactIdjunit/artifactId
+/dependency
+
+dependency
groupIdorg.apache.sqoop/groupId
artifactIdsqoop-spi/artifactId
/dependency
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
--
diff --git
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
index fc12381..7447071 100644
---
a/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
+++
b/connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsExtractor.java
@@ -42,8 +42,6 @@ import java.io.IOException;
* Extract from HDFS.
* Default field delimiter of a record is comma.
*/
-
-
public class HdfsExtractor extends ExtractorConnectionConfiguration,
FromJobConfiguration, HdfsPartition {
public static final Logger LOG = Logger.getLogger(HdfsExtractor.class);
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c0b22b1d/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
--
diff --git
a/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
b/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
new file mode 100644
index 000..8c19d01
--- /dev/null
+++
b/connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/FileUtils.java
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.connector.hdfs;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.util.LinkedList;
+import java.util.List;
+
+public class FileUtils {
+
+ public static boolean exists(String file) throws IOException {
+Path path =
[35/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
--
diff --git a/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
b/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
deleted file mode 100644
index a0a5dc2..000
--- a/shell/src/main/java/org/apache/sqoop/shell/utils/FormFiller.java
+++ /dev/null
@@ -1,939 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.shell.utils;
-
-import jline.ConsoleReader;
-
-import org.apache.commons.cli.CommandLine;
-import org.apache.commons.lang.StringUtils;
-import org.apache.sqoop.common.Direction;
-import org.apache.sqoop.model.MBooleanInput;
-import org.apache.sqoop.model.MLink;
-import org.apache.sqoop.model.MEnumInput;
-import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MInput;
-import org.apache.sqoop.model.MIntegerInput;
-import org.apache.sqoop.model.MMapInput;
-import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MNamedElement;
-import org.apache.sqoop.model.MStringInput;
-import org.apache.sqoop.model.MValidatedElement;
-import org.apache.sqoop.validation.Message;
-import org.apache.sqoop.validation.Status;
-
-import java.io.IOException;
-import java.util.List;
-import java.util.Map;
-import java.util.HashMap;
-import java.util.ResourceBundle;
-
-import static org.apache.sqoop.shell.ShellEnvironment.*;
-
-/**
- * Convenient methods for retrieving user input and CLI options.
- */
-public final class FormFiller {
-
- /**
- * Internal input that will be reused for loading names for link and
- * job objects.
- */
- private static MStringInput nameInput = new MStringInput(object-name,
false, (short)25);
-
- /**
- * Fill job object based on CLI options.
- *
- * @param line Associated console reader object
- * @param job Job that user is suppose to fill in
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillJob(CommandLine line,
-MJob job)
-throws IOException {
-
-job.setName(line.getOptionValue(name));
-
-// Fill in data from user
-return fillForms(line,
- job.getConnectorPart(Direction.FROM).getForms(),
- job.getFrameworkPart().getForms());
- }
-
- /**
- * Fill job object based on user input.
- *
- * @param reader Associated console reader object
- * @param job Job that user is suppose to fill in
- * @param fromConnectorBundle Connector resource bundle
- * @param driverConfigBundle Driver config resource bundle
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillJob(ConsoleReader reader,
-MJob job,
-ResourceBundle fromConnectorBundle,
-ResourceBundle driverConfigBundle,
-ResourceBundle toConnectorBundle)
-throws IOException {
-
-job.setName(getName(reader, job.getName()));
-
-// Fill in data from user
-return fillForms(reader,
- job.getConnectorPart(Direction.FROM).getForms(),
- fromConnectorBundle,
- job.getFrameworkPart().getForms(),
- driverConfigBundle,
- job.getConnectorPart(Direction.TO).getForms(),
- toConnectorBundle);
- }
-
- /**
- * Fill link object based on CLI options.
- *
- * @param line Associated command line options
- * @param link Link that user is suppose to fill in
- * @return True if we filled all inputs, false if user has stopped processing
- * @throws IOException
- */
- public static boolean fillConnection(CommandLine line,
- MLink link)
- throws IOException {
-
-link.setName(line.getOptionValue(name));
-
-// Fill in data from user
-return
[27/50] [abbrv] git commit: SQOOP-1560: Sqoop2: Move matcher out of Schema
SQOOP-1560: Sqoop2: Move matcher out of Schema
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/c1e53e5d
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/c1e53e5d
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/c1e53e5d
Branch: refs/heads/sqoop2
Commit: c1e53e5d54e3e9453aefe74b2dd171ff9b25e5a3
Parents: 97da12a
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Thu Oct 2 16:55:29 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:59:24 2014 -0700
--
.../idf/CSVIntermediateDataFormat.java | 134 -
.../connector/idf/IntermediateDataFormat.java | 9 +-
.../connector/idf/matcher/AbstractMatcher.java | 62 -
.../connector/idf/matcher/LocationMatcher.java | 82 --
.../connector/idf/matcher/NameMatcher.java | 69 -
.../connector/matcher/LocationMatcher.java | 78 ++
.../apache/sqoop/connector/matcher/Matcher.java | 69 +
.../sqoop/connector/matcher/MatcherError.java | 41 +++
.../sqoop/connector/matcher/MatcherFactory.java | 30 ++
.../sqoop/connector/matcher/NameMatcher.java| 74 +
.../idf/TestCSVIntermediateDataFormat.java | 160 ++-
.../org/apache/sqoop/job/mr/SqoopMapper.java| 43 +--
.../job/mr/SqoopOutputFormatLoadExecutor.java | 16 +-
.../java/org/apache/sqoop/job/JobUtils.java | 12 +-
.../org/apache/sqoop/job/TestMapReduce.java | 12 +-
.../java/org/apache/sqoop/job/TestMatching.java | 275 +++
16 files changed, 681 insertions(+), 485 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/c1e53e5d/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
--
diff --git
a/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
b/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
index 2a49221..02d1a51 100644
---
a/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
+++
b/connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
@@ -22,12 +22,7 @@ import com.google.common.annotations.VisibleForTesting;
import org.apache.commons.lang.StringUtils;
import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.connector.idf.matcher.AbstractMatcher;
-import org.apache.sqoop.connector.idf.matcher.LocationMatcher;
-import org.apache.sqoop.connector.idf.matcher.NameMatcher;
import org.apache.sqoop.schema.Schema;
-import org.apache.sqoop.schema.SchemaError;
-import org.apache.sqoop.schema.SchemaMatchOption;
import org.apache.sqoop.schema.type.Column;
import org.apache.sqoop.schema.type.FixedPoint;
import org.apache.sqoop.schema.type.FloatingPoint;
@@ -41,7 +36,6 @@ import java.math.BigDecimal;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
-import java.util.Set;
import java.util.regex.Matcher;
public class CSVIntermediateDataFormat extends IntermediateDataFormatString {
@@ -71,8 +65,7 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
private final ListInteger stringFieldIndices = new ArrayListInteger();
private final ListInteger byteFieldIndices = new ArrayListInteger();
- private Schema fromSchema;
- private Schema toSchema;
+ private Schema schema;
/**
* {@inheritDoc}
@@ -94,11 +87,11 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
* {@inheritDoc}
*/
@Override
- public void setFromSchema(Schema schema) {
+ public void setSchema(Schema schema) {
if(schema == null) {
return;
}
-this.fromSchema = schema;
+this.schema = schema;
ListColumn columns = schema.getColumns();
int i = 0;
for(Column col : columns) {
@@ -112,19 +105,6 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
}
/**
- * {@inheritDoc}
- */
- @Override
- public void setToSchema(Schema schema) {
-if(schema == null) {
- return;
-}
-this.toSchema = schema;
- }
-
-
-
- /**
* Custom CSV parser that honors quoting and escaped quotes.
* All other escaping is handled elsewhere.
*
@@ -180,69 +160,68 @@ public class CSVIntermediateDataFormat extends
IntermediateDataFormatString {
/**
* {@inheritDoc}
- *
- * The CSV data is ordered according to the fromSchema. We translate it to
the TO schema.
- * We currently have 3 methods of matching fields in one schema to another:
- * - by location
- * - by name
- * - user-defined matching
- *
- * If one schema
[03/50] [abbrv] git commit: SQOOP-1468: Sqoop2: Validations: Introduce advanced validators that can be parametrized
SQOOP-1468: Sqoop2: Validations: Introduce advanced validators that can be
parametrized
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/e71dd757
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/e71dd757
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/e71dd757
Branch: refs/heads/sqoop2
Commit: e71dd75793982b1d6e5109b8ef2209c835a79b68
Parents: c0b22b1
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Wed Sep 3 11:17:48 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:17 2014 -0700
--
.../apache/sqoop/model/ConfigurationClass.java | 4 +-
.../java/org/apache/sqoop/model/FormClass.java | 4 +-
.../main/java/org/apache/sqoop/model/Input.java | 4 +-
.../java/org/apache/sqoop/model/Validator.java | 45
.../sqoop/validation/ValidationResult.java | 4 +-
.../sqoop/validation/ValidationRunner.java | 18 ++--
.../validators/AbstractValidator.java | 103 +++
.../validation/validators/ClassAvailable.java | 2 +-
.../sqoop/validation/validators/Contains.java | 32 ++
.../sqoop/validation/validators/NotEmpty.java | 2 +-
.../sqoop/validation/validators/NotNull.java| 2 +-
.../sqoop/validation/validators/StartsWith.java | 32 ++
.../sqoop/validation/validators/Validator.java | 80 --
.../sqoop/validation/TestValidationRunner.java | 42 ++--
.../validators/TestClassAvailable.java | 2 +-
.../validation/validators/TestContains.java | 90
.../validation/validators/TestNotEmpty.java | 2 +-
.../validation/validators/TestNotNull.java | 2 +-
.../validation/validators/TestStartsWith.java | 74 +
.../validation/validators/TestValidator.java| 2 +-
.../jdbc/configuration/ConnectionForm.java | 12 ++-
.../jdbc/configuration/FromTableForm.java | 19 ++--
.../jdbc/configuration/ToTableForm.java | 7 +-
23 files changed, 450 insertions(+), 134 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
b/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
index 5323bd9..73374d8 100644
--- a/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
+++ b/common/src/main/java/org/apache/sqoop/model/ConfigurationClass.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -37,5 +35,5 @@ public @interface ConfigurationClass {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/FormClass.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/FormClass.java
b/common/src/main/java/org/apache/sqoop/model/FormClass.java
index 48bff3c..6048d03 100644
--- a/common/src/main/java/org/apache/sqoop/model/FormClass.java
+++ b/common/src/main/java/org/apache/sqoop/model/FormClass.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -43,5 +41,5 @@ public @interface FormClass {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e71dd757/common/src/main/java/org/apache/sqoop/model/Input.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/Input.java
b/common/src/main/java/org/apache/sqoop/model/Input.java
index 61fc01a..2b32e43 100644
--- a/common/src/main/java/org/apache/sqoop/model/Input.java
+++ b/common/src/main/java/org/apache/sqoop/model/Input.java
@@ -17,8 +17,6 @@
*/
package org.apache.sqoop.model;
-import org.apache.sqoop.validation.validators.Validator;
-
import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
@@ -51,5 +49,5 @@ public @interface Input {
*
* @return
*/
- Class? extends Validator[] validators() default {};
+ Validator[] validators() default {};
}
[24/50] [abbrv] SQOOP-1478: Sqoop2: Migrate remaining tests from junit 3 to 4
http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
index 7378eda..8dd1ce2 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestLinkHandling.java
@@ -22,11 +22,15 @@ import org.apache.sqoop.model.MLink;
import org.apache.sqoop.model.MForm;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
+import org.junit.Before;
+import org.junit.Test;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import static org.junit.Assert.*;
+
/**
* Test link methods on Derby repository.
*/
@@ -34,7 +38,7 @@ public class TestLinkHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
- @Override
+ @Before
public void setUp() throws Exception {
super.setUp();
@@ -47,6 +51,7 @@ public class TestLinkHandling extends DerbyTestCase {
loadConnectorAndDriverConfig();
}
+ @Test
public void testFindLink() throws Exception {
// Let's try to find non existing link
try {
@@ -81,6 +86,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertNull(forms.get(1).getInputs().get(1).getValue());
}
+ @Test
public void testFindLinks() throws Exception {
ListMLink list;
@@ -98,6 +104,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(CB, list.get(1).getName());
}
+ @Test
public void testExistsLink() throws Exception {
// There shouldn't be anything on empty repository
assertFalse(handler.existsLink(1, getDerbyDatabaseConnection()));
@@ -111,6 +118,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertFalse(handler.existsLink(3, getDerbyDatabaseConnection()));
}
+ @Test
public void testCreateLink() throws Exception {
MLink link = getLink();
@@ -150,6 +158,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertCountForTable(SQOOP.SQ_CONNECTION_INPUT, 8);
}
+ @Test
public void testInUseLink() throws Exception {
loadLinks();
@@ -160,6 +169,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertTrue(handler.inUseLink(1, getDerbyDatabaseConnection()));
}
+ @Test
public void testUpdateLink() throws Exception {
loadLinks();
@@ -205,6 +215,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(((Map)forms.get(1).getInputs().get(1).getValue()).size(), 0);
}
+ @Test
public void testEnableAndDisableLink() throws Exception {
loadLinks();
@@ -223,6 +234,7 @@ public class TestLinkHandling extends DerbyTestCase {
assertEquals(true, retrieved.getEnabled());
}
+ @Test
public void testDeleteLink() throws Exception {
loadLinks();
http://git-wip-us.apache.org/repos/asf/sqoop/blob/97da12aa/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
index 38a995f..cf0944d 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestSubmissionHandling.java
@@ -22,11 +22,15 @@ import org.apache.sqoop.submission.SubmissionStatus;
import org.apache.sqoop.submission.counter.Counter;
import org.apache.sqoop.submission.counter.CounterGroup;
import org.apache.sqoop.submission.counter.Counters;
+import org.junit.Before;
+import org.junit.Test;
import java.util.Calendar;
import java.util.Date;
import java.util.List;
+import static org.junit.Assert.*;
+
/**
*
*/
@@ -34,7 +38,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
- @Override
+ @Before
public void setUp() throws Exception {
super.setUp();
@@ -53,6 +57,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
loadJobs();
}
+ @Test
public void testFindSubmissionsUnfinished() throws Exception {
ListMSubmission submissions;
@@ -67,6 +72,7 @@ public class TestSubmissionHandling extends DerbyTestCase {
assertEquals(2, submissions.size());
}
+ @Test
public void testExistsSubmission() throws Exception {
// There shouldn't be anything on empty repository
[16/50] [abbrv] SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
http://git-wip-us.apache.org/repos/asf/sqoop/blob/049994a0/common/src/test/java/org/apache/sqoop/json/TestConnectionBean.java
--
diff --git a/common/src/test/java/org/apache/sqoop/json/TestConnectionBean.java
b/common/src/test/java/org/apache/sqoop/json/TestConnectionBean.java
deleted file mode 100644
index 19f81a8..000
--- a/common/src/test/java/org/apache/sqoop/json/TestConnectionBean.java
+++ /dev/null
@@ -1,138 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * License); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an AS IS BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.sqoop.json;
-
-import org.apache.sqoop.model.MConnection;
-import org.apache.sqoop.model.MStringInput;
-import org.json.simple.JSONArray;
-import org.json.simple.JSONObject;
-import org.json.simple.JSONArray;
-import org.json.simple.JSONValue;
-import org.junit.Test;
-
-import java.util.Date;
-
-import static junit.framework.Assert.*;
-import static org.apache.sqoop.json.TestUtil.*;
-
-/**
- *
- */
-public class TestConnectionBean {
- @Test
- public void testSerialization() {
-Date created = new Date();
-Date updated = new Date();
-MConnection connection = getConnection(ahoj);
-connection.setName(Connection);
-connection.setPersistenceId(666);
-connection.setCreationUser(admin);
-connection.setCreationDate(created);
-connection.setLastUpdateUser(user);
-connection.setLastUpdateDate(updated);
-connection.setEnabled(false);
-
-// Fill some data at the beginning
-MStringInput input = (MStringInput)
connection.getConnectorPart().getForms()
- .get(0).getInputs().get(0);
-input.setValue(Hi there!);
-
-// Serialize it to JSON object
-ConnectionBean bean = new ConnectionBean(connection);
-JSONObject json = bean.extract(false);
-
-// Check for sensitivity
-JSONArray all = (JSONArray)json.get(all);
-JSONObject allItem = (JSONObject)all.get(0);
-JSONArray connectors = (JSONArray)allItem.get(connector);
-JSONObject connector = (JSONObject)connectors.get(0);
-JSONArray inputs = (JSONArray)connector.get(inputs);
-for (Object input1 : inputs) {
- assertTrue(((JSONObject)input1).containsKey(sensitive));
-}
-
-// Move it across network in text form
-String string = json.toJSONString();
-
-// Retrieved transferred object
-JSONObject retrievedJson = (JSONObject) JSONValue.parse(string);
-ConnectionBean retrievedBean = new ConnectionBean();
-retrievedBean.restore(retrievedJson);
-MConnection target = retrievedBean.getConnections().get(0);
-
-// Check id and name
-assertEquals(666, target.getPersistenceId());
-assertEquals(Connection, target.getName());
-assertEquals(admin, target.getCreationUser());
-assertEquals(created, target.getCreationDate());
-assertEquals(user, target.getLastUpdateUser());
-assertEquals(updated, target.getLastUpdateDate());
-assertEquals(false, target.getEnabled());
-
-// Test that value was correctly moved
-MStringInput targetInput = (MStringInput) target.getConnectorPart()
- .getForms().get(0).getInputs().get(0);
-assertEquals(Hi there!, targetInput.getValue());
- }
-
- @Test
- public void testSensitivityFilter() {
-Date created = new Date();
-Date updated = new Date();
-MConnection connection = getConnection(ahoj);
-connection.setName(Connection);
-connection.setPersistenceId(666);
-connection.setCreationUser(admin);
-connection.setCreationDate(created);
-connection.setLastUpdateUser(user);
-connection.setLastUpdateDate(updated);
-connection.setEnabled(true);
-
-// Fill some data at the beginning
-MStringInput input = (MStringInput)
connection.getConnectorPart().getForms()
- .get(0).getInputs().get(0);
-input.setValue(Hi there!);
-
-// Serialize it to JSON object
-ConnectionBean bean = new ConnectionBean(connection);
-JSONObject json = bean.extract(false);
-JSONObject jsonFiltered = bean.extract(true);
-
-// Sensitive values should exist
-JSONArray all = (JSONArray)json.get(all);
-JSONObject allItem = (JSONObject)all.get(0);
-JSONArray connectors = (JSONArray)allItem.get(connector);
-JSONObject connector =
[18/50] [abbrv] git commit: SQOOP-1497: Sqoop2: Entity Nomenclature Revisited
SQOOP-1497: Sqoop2: Entity Nomenclature Revisited Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/049994a0 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/049994a0 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/049994a0 Branch: refs/heads/sqoop2 Commit: 049994a0217164b703e754978a85a6c17fcb720a Parents: b04e796 Author: Veena Basavaraj veena.b...@gmail.com Authored: Tue Sep 23 15:28:24 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 17:58:18 2014 -0700 -- .../org/apache/sqoop/client/SqoopClient.java| 280 .../sqoop/client/request/ConnectionRequest.java | 91 --- .../sqoop/client/request/ConnectorRequest.java | 47 -- .../request/ConnectorResourceRequest.java | 47 ++ .../request/DriverConfigResourceRequest.java| 42 ++ .../sqoop/client/request/FrameworkRequest.java | 42 -- .../apache/sqoop/client/request/JobRequest.java | 91 --- .../client/request/JobResourceRequest.java | 91 +++ .../client/request/LinkResourceRequest.java | 89 +++ .../apache/sqoop/client/request/Request.java| 110 --- .../sqoop/client/request/ResourceRequest.java | 113 +++ .../sqoop/client/request/SqoopRequests.java | 149 .../client/request/SqoopResourceRequests.java | 149 .../sqoop/client/request/SubmissionRequest.java | 80 --- .../request/SubmissionResourceRequest.java | 80 +++ .../sqoop/client/request/VersionRequest.java| 35 - .../client/request/VersionResourceRequest.java | 35 + .../apache/sqoop/client/TestSqoopClient.java| 105 ++- .../org/apache/sqoop/common/SqoopException.java | 2 +- .../org/apache/sqoop/etl/io/DataReader.java | 10 +- .../org/apache/sqoop/job/etl/ActorContext.java | 77 -- .../apache/sqoop/job/etl/DestroyerContext.java | 2 +- .../apache/sqoop/job/etl/ExtractorContext.java | 2 +- .../sqoop/job/etl/InitializerContext.java | 2 +- .../org/apache/sqoop/job/etl/LoaderContext.java | 2 +- .../sqoop/job/etl/PartitionerContext.java | 2 +- .../sqoop/job/etl/TransferableContext.java | 76 ++ .../org/apache/sqoop/json/ConnectionBean.java | 185 - .../sqoop/json/ConnectionValidationBean.java| 143 .../org/apache/sqoop/json/ConnectorBean.java| 37 +- .../org/apache/sqoop/json/DriverConfigBean.java | 92 +++ .../org/apache/sqoop/json/FrameworkBean.java| 102 --- .../java/org/apache/sqoop/json/JobBean.java | 63 +- .../java/org/apache/sqoop/json/LinkBean.java| 184 + .../apache/sqoop/json/LinkValidationBean.java | 143 .../sqoop/json/util/FormSerialization.java | 2 +- .../json/util/ResourceBundleSerialization.java | 13 +- .../main/java/org/apache/sqoop/model/Input.java | 2 +- .../org/apache/sqoop/model/MConnection.java | 130 .../org/apache/sqoop/model/MDriverConfig.java | 97 +++ .../java/org/apache/sqoop/model/MFramework.java | 99 --- .../java/org/apache/sqoop/model/MInput.java | 2 +- .../main/java/org/apache/sqoop/model/MJob.java | 34 +- .../main/java/org/apache/sqoop/model/MLink.java | 129 .../apache/sqoop/json/TestConnectionBean.java | 138 .../apache/sqoop/json/TestDriverConfigBean.java | 64 ++ .../apache/sqoop/json/TestFrameworkBean.java| 64 -- .../java/org/apache/sqoop/json/TestJobBean.java | 4 +- .../org/apache/sqoop/json/TestLinkBean.java | 138 .../java/org/apache/sqoop/json/TestUtil.java| 23 +- .../apache/sqoop/json/TestValidationBean.java | 12 +- .../sqoop/model/TestMAccountableEntity.java | 24 +- .../org/apache/sqoop/model/TestMConnection.java | 122 .../apache/sqoop/model/TestMDriverConfig.java | 47 ++ .../org/apache/sqoop/model/TestMFramework.java | 47 -- .../java/org/apache/sqoop/model/TestMLink.java | 122 .../connector/jdbc/GenericJdbcConnector.java| 12 +- .../jdbc/GenericJdbcConnectorConstants.java | 2 +- .../GenericJdbcConnectorMetadataUpgrader.java | 83 --- .../jdbc/GenericJdbcConnectorUpgrader.java | 83 +++ .../connector/jdbc/GenericJdbcExtractor.java| 15 +- .../jdbc/GenericJdbcFromDestroyer.java | 6 +- .../jdbc/GenericJdbcFromInitializer.java| 60 +- .../sqoop/connector/jdbc/GenericJdbcLoader.java | 14 +- .../connector/jdbc/GenericJdbcPartitioner.java | 8 +- .../connector/jdbc/GenericJdbcToDestroyer.java | 22 +- .../jdbc/GenericJdbcToInitializer.java | 52 +- .../connector/jdbc/GenericJdbcValidator.java| 66 +- .../configuration/ConnectionConfiguration.java | 34 - .../jdbc/configuration/ConnectionForm.java | 64 -- .../jdbc/configuration/FromJobConfig.java | 68 ++ .../configuration/FromJobConfiguration.java | 4 +- .../jdbc/configuration/FromTableForm.java | 68 -- .../jdbc/configuration/LinkConfig.java | 64 ++
[33/50] [abbrv] git commit: SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector
SQOOP-1481: SQOOP2: Document the public apis and end-end design for the SQ2 Connector (Abraham Elmahrek via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/b6000365 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/b6000365 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/b6000365 Branch: refs/heads/sqoop2 Commit: b6000365736d7b03124417f6b8731d8760011432 Parents: 4c964a9 Author: Jarek Jarcec Cecho jar...@apache.org Authored: Thu Oct 9 08:16:52 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 18:22:51 2014 -0700 -- docs/src/site/sphinx/CommandLineClient.rst | 6 +- docs/src/site/sphinx/ConnectorDevelopment.rst | 154 ++- docs/src/site/sphinx/RESTAPI.rst| 54 +-- .../sqoop/connector/spi/SqoopConnector.java | 2 +- 4 files changed, 95 insertions(+), 121 deletions(-) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/b6000365/docs/src/site/sphinx/CommandLineClient.rst -- diff --git a/docs/src/site/sphinx/CommandLineClient.rst b/docs/src/site/sphinx/CommandLineClient.rst index fb7f780..7cc6449 100644 --- a/docs/src/site/sphinx/CommandLineClient.rst +++ b/docs/src/site/sphinx/CommandLineClient.rst @@ -329,14 +329,14 @@ Create new job object. ++--+ | Argument | Description | ++==+ -| ``-x``, ``--xid x`` | Create new job object for connection with id ``x`` | +| ``-f``, ``--from x`` | Create new job object with a FROM connection with id ``x`` | ++--+ -| ``-t``, ``--type t`` | Create new job object with type ``t`` (``import``, ``export``) | +| ``-t``, ``--to t`` | Create new job object with a TO connection with id ``x`` | ++--+ Example: :: - create job --xid 1 + create job --from 1 --to 2 Update Command -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/b6000365/docs/src/site/sphinx/ConnectorDevelopment.rst -- diff --git a/docs/src/site/sphinx/ConnectorDevelopment.rst b/docs/src/site/sphinx/ConnectorDevelopment.rst index ae4f721..d700e4c 100644 --- a/docs/src/site/sphinx/ConnectorDevelopment.rst +++ b/docs/src/site/sphinx/ConnectorDevelopment.rst @@ -26,17 +26,15 @@ using the code of built-in connector ( ``GenericJdbcConnector`` ) as example. What is Connector? ++ -Connector provides interaction with external databases. -Connector reads data from databases for import, -and write data to databases for export. -Interaction with Hadoop is taken cared by common modules of Sqoop 2 framework. +The connector provides the facilities to interact with external data sources. +The connector can read from, or write to, a data source. When do we add a new connector? === You add a new connector when you need to extract data from a new data source, or load data to a new target. In addition to the connector API, Sqoop 2 also has an engine interface. -At the moment the only engine is MapReduce,but we may support additional engines in the future. +At the moment the only engine is MapReduce, but we may support additional engines in the future. Since many parallel execution engines are capable of reading/writing data there may be a question of whether support for specific data stores should be done through a new connector or new engine. @@ -51,57 +49,73 @@ Connector Implementation The ``SqoopConnector`` class defines functionality which must be provided by Connectors. -Each Connector must extends ``SqoopConnector`` and overrides methods shown below. +Each Connector must extend ``SqoopConnector`` and override the methods shown below. :: public abstract String getVersion(); public abstract ResourceBundle getBundle(Locale locale); - public abstract Class getConnectionConfigurationClass(); - public abstract Class getJobConfigurationClass(MJob.Type jobType); - public abstract Importer getImporter(); - public abstract Exporter getExporter(); + public abstract Class getLinkConfigurationClass(); + public abstract Class getJobConfigurationClass(Direction direction); + public abstract From getFrom(); + public abstract To getTo(); public abstract Validator getValidator(); public abstract MetadataUpgrader
[21/50] [abbrv] git commit: SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/3d539dd4
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/3d539dd4
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/3d539dd4
Branch: refs/heads/sqoop2
Commit: 3d539dd4d7477324dfe62a4e57f684351769b000
Parents: af25bcc
Author: Abraham Elmahrek abra...@elmahrek.com
Authored: Fri Sep 19 16:24:59 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:58:18 2014 -0700
--
.../org/apache/sqoop/json/SubmissionBean.java | 20 +-
.../org/apache/sqoop/model/MSubmission.java | 31 +-
.../apache/sqoop/json/TestSubmissionBean.java | 12 +-
.../sqoop/connector/jdbc/TestToInitializer.java | 1 -
.../idf/CSVIntermediateDataFormat.java | 4 -
.../connector/idf/IntermediateDataFormat.java | 4 -
.../idf/CSVIntermediateDataFormatTest.java | 222 --
.../idf/TestCSVIntermediateDataFormat.java | 222 ++
.../sqoop/connector/ConnectorManager.java | 5 +-
.../apache/sqoop/framework/ExecutionEngine.java | 20 +-
.../org/apache/sqoop/framework/JobManager.java | 447 ++--
.../org/apache/sqoop/framework/JobRequest.java | 356 ++
.../sqoop/framework/SubmissionEngine.java | 7 +-
.../sqoop/framework/SubmissionRequest.java | 361 --
.../sqoop/framework/TestFrameworkValidator.java | 182 +++--
.../apache/sqoop/framework/TestJobManager.java | 173 +
.../apache/sqoop/framework/TestJobRequest.java | 71 ++
.../sqoop/framework/TestSubmissionRequest.java | 71 --
.../sqoop/repository/TestJdbcRepository.java| 694 +--
.../sqoop/execution/mapreduce/MRJobRequest.java | 102 +++
.../mapreduce/MRSubmissionRequest.java | 102 ---
.../mapreduce/MapreduceExecutionEngine.java | 51 +-
.../apache/sqoop/job/mr/ConfigurationUtils.java | 3 -
.../apache/sqoop/job/mr/ProgressRunnable.java | 4 +-
.../sqoop/job/mr/SqoopDestroyerExecutor.java| 7 +-
.../sqoop/job/mr/SqoopFileOutputFormat.java | 4 +-
.../org/apache/sqoop/job/mr/SqoopMapper.java| 50 +-
.../job/mr/SqoopOutputFormatLoadExecutor.java | 28 +-
.../sqoop/shell/utils/SubmissionDisplayer.java | 8 +-
.../org/apache/sqoop/job/etl/CallbackBase.java | 49 --
.../java/org/apache/sqoop/job/etl/From.java | 2 +-
.../main/java/org/apache/sqoop/job/etl/To.java | 2 +-
.../org/apache/sqoop/job/etl/Transferable.java | 51 ++
.../org/apache/sqoop/validation/Validator.java | 1 -
.../mapreduce/MapreduceSubmissionEngine.java| 12 +-
35 files changed, 1780 insertions(+), 1599 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/3d539dd4/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
--
diff --git a/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
b/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
index 61d6576..9b1ae74 100644
--- a/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
+++ b/common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
@@ -52,8 +52,8 @@ public class SubmissionBean implements JsonBean {
private static final String EXCEPTION_TRACE = exception-trace;
private static final String PROGRESS = progress;
private static final String COUNTERS = counters;
- private static final String CONNECTOR_SCHEMA = schema-connector;
- private static final String HIO_SCHEMA = schema-hio;
+ private static final String FROM_SCHEMA = schema-from;
+ private static final String TO_SCHEMA = schema-to;
private ListMSubmission submissions;
@@ -116,11 +116,11 @@ public class SubmissionBean implements JsonBean {
if(submission.getCounters() != null) {
object.put(COUNTERS, extractCounters(submission.getCounters()));
}
- if(submission.getConnectorSchema() != null) {
-object.put(CONNECTOR_SCHEMA,
extractSchema(submission.getConnectorSchema()));
+ if(submission.getFromSchema() != null) {
+object.put(FROM_SCHEMA, extractSchema(submission.getFromSchema()));
}
- if(submission.getHioSchema() != null) {
-object.put(HIO_SCHEMA, extractSchema(submission.getHioSchema()));
+ if(submission.getToSchema() != null) {
+object.put(TO_SCHEMA, extractSchema(submission.getToSchema()));
}
array.add(object);
@@ -188,11 +188,11 @@ public class SubmissionBean implements JsonBean {
if(object.containsKey(COUNTERS)) {
submission.setCounters(restoreCounters((JSONObject)
object.get(COUNTERS)));
}
- if(object.containsKey(CONNECTOR_SCHEMA)) {
-submission.setConnectorSchema(restoreSchemna((JSONObject)
[30/50] [abbrv] git commit: SQOOP-1479: Sqoop2: Validations: Cache Validator objects in ValidationRunner
SQOOP-1479: Sqoop2: Validations: Cache Validator objects in ValidationRunner
(Jarek Jarcec Cecho via Abraham Elmahrek)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/e540668b
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/e540668b
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/e540668b
Branch: refs/heads/sqoop2
Commit: e540668b5569753165d812522e1c5bc24cc1fe5b
Parents: c1e53e5
Author: Abraham Elmahrek abra...@elmahrek.com
Authored: Thu Oct 2 21:26:41 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 17:59:25 2014 -0700
--
.../sqoop/validation/ValidationError.java | 3 ++
.../sqoop/validation/ValidationRunner.java | 35 ++--
2 files changed, 35 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e540668b/common/src/main/java/org/apache/sqoop/validation/ValidationError.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/validation/ValidationError.java
b/common/src/main/java/org/apache/sqoop/validation/ValidationError.java
index 264d6e3..ec64f10 100644
--- a/common/src/main/java/org/apache/sqoop/validation/ValidationError.java
+++ b/common/src/main/java/org/apache/sqoop/validation/ValidationError.java
@@ -31,6 +31,9 @@ public enum ValidationError implements ErrorCode {
VALIDATION_0002(Usage of missing field),
VALIDATION_0003(Invalid representation of form and input field),
+
+ VALIDATION_0004(Can't find validator class),
+
;
private final String message;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/e540668b/common/src/main/java/org/apache/sqoop/validation/ValidationRunner.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/validation/ValidationRunner.java
b/common/src/main/java/org/apache/sqoop/validation/ValidationRunner.java
index f36faf2..8ffc0d4 100644
--- a/common/src/main/java/org/apache/sqoop/validation/ValidationRunner.java
+++ b/common/src/main/java/org/apache/sqoop/validation/ValidationRunner.java
@@ -17,6 +17,7 @@
*/
package org.apache.sqoop.validation;
+import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.model.ConfigurationClass;
import org.apache.sqoop.model.Form;
import org.apache.sqoop.model.FormClass;
@@ -27,6 +28,8 @@ import org.apache.sqoop.utils.ClassUtils;
import org.apache.sqoop.validation.validators.AbstractValidator;
import java.lang.reflect.Field;
+import java.util.HashMap;
+import java.util.Map;
/**
* Validation runner that will run validators associated with given
configuration
@@ -39,12 +42,23 @@ import java.lang.reflect.Field;
* Which means that form validator don't have to repeat it's input validators
as it will
* be never called if the input's are not valid. Similarly Class validators
won't be called
* unless all forms will pass validators.
- *
- * TODO: Cache the validators instances, so that we don't have create new
instance every time
*/
public class ValidationRunner {
/**
+ * Private cache of instantiated validators.
+ *
+ * We're expecting that this cache will be very small as the number of
possible validators
+ * is driven to high extent by the number of connectors and hence we don't
have a cache
+ * eviction at the moment.
+ */
+ private MapClass? extends AbstractValidator, AbstractValidator cache;
+
+ public ValidationRunner() {
+cache = new HashMapClass? extends AbstractValidator,
AbstractValidator();
+ }
+
+ /**
* Validate given configuration instance.
*
* @param config Configuration instance
@@ -137,7 +151,22 @@ public class ValidationRunner {
* @return
*/
private AbstractValidator executeValidator(Object object, Validator
validator) {
-AbstractValidator instance = (AbstractValidator)
ClassUtils.instantiate(validator.value());
+// Try to get validator instance from the cache
+AbstractValidator instance = cache.get(validator.value());
+
+if(instance == null) {
+ instance = (AbstractValidator) ClassUtils.instantiate(validator.value());
+
+ // This could happen if we would be missing some connector's jars on our
classpath
+ if(instance == null) {
+throw new SqoopException(ValidationError.VALIDATION_0004,
validator.value().getName());
+ }
+
+ cache.put(validator.value(), instance);
+} else {
+ instance.reset();
+}
+
instance.setStringArgument(validator.strArg());
instance.validate(object);
return instance;
[37/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
--
diff --git
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
index 47350ea..595b1c8 100644
---
a/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
+++
b/repository/repository-derby/src/test/java/org/apache/sqoop/repository/derby/TestJobHandling.java
@@ -17,9 +17,14 @@
*/
package org.apache.sqoop.repository.derby;
+import java.sql.Connection;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.SqoopException;
-import org.apache.sqoop.model.MForm;
+import org.apache.sqoop.model.MConfig;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
@@ -38,18 +43,19 @@ import static org.junit.Assert.*;
public class TestJobHandling extends DerbyTestCase {
DerbyRepositoryHandler handler;
+ Connection derbyConnection;
@Before
public void setUp() throws Exception {
super.setUp();
+derbyConnection = getDerbyDatabaseConnection();
handler = new DerbyRepositoryHandler();
// We always needs schema for this test case
createSchema();
-// We always needs connector and framework structures in place
-loadConnectorAndDriverConfig();
+loadConnectorLinkConfig();
// We always needs connection metadata in place
loadLinks();
@@ -59,58 +65,53 @@ public class TestJobHandling extends DerbyTestCase {
public void testFindJob() throws Exception {
// Let's try to find non existing job
try {
- handler.findJob(1, getDerbyDatabaseConnection());
+ handler.findJob(1, derbyConnection);
fail();
} catch(SqoopException ex) {
assertEquals(DerbyRepoError.DERBYREPO_0030, ex.getErrorCode());
}
-// Load prepared connections into database
loadJobs();
-MJob jobImport = handler.findJob(1, getDerbyDatabaseConnection());
-assertNotNull(jobImport);
-assertEquals(1, jobImport.getPersistenceId());
-assertEquals(JA, jobImport.getName());
-
-ListMForm forms;
-
-// Check connector parts
-forms = jobImport.getConnectorPart(Direction.FROM).getForms();
-assertEquals(2, forms.size());
-assertEquals(Value5, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value5, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
-
-forms = jobImport.getConnectorPart(Direction.TO).getForms();
-assertEquals(2, forms.size());
-assertEquals(Value9, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value9, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
-
-// Check framework part
-forms = jobImport.getFrameworkPart().getForms();
-assertEquals(2, forms.size());
-assertEquals(Value17, forms.get(0).getInputs().get(0).getValue());
-assertNull(forms.get(0).getInputs().get(1).getValue());
-assertEquals(Value19, forms.get(1).getInputs().get(0).getValue());
-assertNull(forms.get(1).getInputs().get(1).getValue());
+MJob firstJob = handler.findJob(1, derbyConnection);
+assertNotNull(firstJob);
+assertEquals(1, firstJob.getPersistenceId());
+assertEquals(JA, firstJob.getName());
+
+ListMConfig configs;
+
+configs = firstJob.getJobConfig(Direction.FROM).getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value5, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value5, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
+
+configs = firstJob.getJobConfig(Direction.TO).getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value9, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value9, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
+
+configs = firstJob.getDriverConfig().getConfigs();
+assertEquals(2, configs.size());
+assertEquals(Value13, configs.get(0).getInputs().get(0).getValue());
+assertNull(configs.get(0).getInputs().get(1).getValue());
+assertEquals(Value15, configs.get(1).getInputs().get(0).getValue());
+assertNull(configs.get(1).getInputs().get(1).getValue());
}
@Test
public void
[43/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/common/src/test/java/org/apache/sqoop/model/TestMConnector.java
--
diff --git a/common/src/test/java/org/apache/sqoop/model/TestMConnector.java
b/common/src/test/java/org/apache/sqoop/model/TestMConnector.java
index d8bc94d..89ae440 100644
--- a/common/src/test/java/org/apache/sqoop/model/TestMConnector.java
+++ b/common/src/test/java/org/apache/sqoop/model/TestMConnector.java
@@ -17,6 +17,13 @@
*/
package org.apache.sqoop.model;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertNull;
+import static org.junit.Assert.assertTrue;
+
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
@@ -24,39 +31,34 @@ import java.util.List;
import org.apache.sqoop.common.Direction;
import org.junit.Test;
-import static org.junit.Assert.*;
-
-/**
- * Test class for org.apache.sqoop.model.TestMConnector
- */
public class TestMConnector {
private MConnector createConnector(ListDirection supportedDirections) {
-ListMForm forms = new ArrayListMForm();
-MIntegerInput input = new MIntegerInput(INTEGER-INPUT, false);
-input.setValue(100);
+ListMConfig configs = new ArrayListMConfig();
+MIntegerInput inputs = new MIntegerInput(INTEGER-INPUT, false);
+inputs.setValue(100);
MStringInput strInput = new MStringInput(STRING-INPUT,false,(short)20);
strInput.setValue(TEST-VALUE);
ListMInput? list = new ArrayListMInput?();
-list.add(input);
+list.add(inputs);
list.add(strInput);
-MForm form = new MForm(FORMNAME, list);
-forms.add(form);
+MConfig config = new MConfig(CONFIGNAME, list);
+configs.add(config);
-MConnectionForms connectionForms1 = new MConnectionForms(forms);
-MJobForms fromForm = null;
-MJobForms toForm = null;
+MLinkConfig linkConfig = new MLinkConfig(configs);
+MFromConfig fromConfig = null;
+MToConfig toConfig = null;
if (supportedDirections.contains(Direction.FROM)) {
- fromForm = new MJobForms(forms);
+ fromConfig = new MFromConfig(configs);
}
if (supportedDirections.contains(Direction.TO)) {
- toForm = new MJobForms(forms);
+ toConfig = new MToConfig(configs);
}
return new MConnector(NAME, CLASSNAME, 1.0,
-connectionForms1, fromForm, toForm);
+linkConfig, fromConfig, toConfig);
}
/**
@@ -64,32 +66,32 @@ public class TestMConnector {
*/
@Test
public void testInitialization() {
-ListMForm fromJobForms = new ArrayListMForm();
-ListMForm toJobForms = new ArrayListMForm();
-MConnectionForms connectionForms1 = new MConnectionForms(fromJobForms);
-MJobForms fromJobForm1 = new MJobForms(fromJobForms);
-MJobForms toJobForm1 = new MJobForms(toJobForms);
+ListMConfig fromJobConfig = new ArrayListMConfig();
+ListMConfig toJobConfig = new ArrayListMConfig();
+MLinkConfig linkConfig = new MLinkConfig(fromJobConfig);
+MFromConfig fromConfig1 = new MFromConfig(fromJobConfig);
+MToConfig toConfig1 = new MToConfig(toJobConfig);
MConnector connector1 = new MConnector(NAME, CLASSNAME, 1.0,
-connectionForms1, fromJobForm1, toJobForm1);
+linkConfig, fromConfig1, toConfig1);
assertEquals(NAME, connector1.getUniqueName());
assertEquals(CLASSNAME, connector1.getClassName());
assertEquals(1.0, connector1.getVersion());
MConnector connector2 = new MConnector(NAME, CLASSNAME, 1.0,
-connectionForms1, fromJobForm1, toJobForm1);
+linkConfig, fromConfig1, toConfig1);
assertEquals(connector2, connector1);
MConnector connector3 = new MConnector(NAME1, CLASSNAME, 2.0,
-connectionForms1, fromJobForm1, toJobForm1);
+linkConfig, fromConfig1, toConfig1);
assertFalse(connector1.equals(connector3));
try {
- connector1 = new MConnector(null, CLASSNAME, 1.0, connectionForms1,
- fromJobForm1, toJobForm1); // Expecting null pointer exception
+ connector1 = new MConnector(null, CLASSNAME, 1.0, linkConfig,
+ fromConfig1, toConfig1); // Expecting null pointer exception
} catch (NullPointerException e) {
assertTrue(true);
}
try {
- connector1 = new MConnector(NAME, null, 1.0, connectionForms1,
- fromJobForm1, toJobForm1); // Expecting null pointer exception
+ connector1 = new MConnector(NAME, null, 1.0, linkConfig,
+ fromConfig1, toConfig1); // Expecting null pointer exception
} catch (NullPointerException e) {
assertTrue(true);
}
@@ -97,48 +99,48 @@ public class TestMConnector {
@Test
public void testClone() {
-MConnector connector1 = createConnector(Arrays.asList(Direction.FROM,
Direction.TO));
-
[45/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/common/src/main/java/org/apache/sqoop/model/MConnector.java
--
diff --git a/common/src/main/java/org/apache/sqoop/model/MConnector.java
b/common/src/main/java/org/apache/sqoop/model/MConnector.java
index 7999b08..2f42191 100644
--- a/common/src/main/java/org/apache/sqoop/model/MConnector.java
+++ b/common/src/main/java/org/apache/sqoop/model/MConnector.java
@@ -23,28 +23,27 @@ import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.common.SupportedDirections;
/**
- * Connector metadata.
- *
- * Includes unique id that identifies connector in metadata store, unique human
- * readable name, corresponding name and all forms for all supported job types.
+ * Connector entity supports the FROM/TO {@link Transferable} Includes unique
id
+ * that identifies connector in the repository, unique human readable name,
+ * corresponding name and all configs to support the from and to data sources
*/
public final class MConnector extends MPersistableEntity implements MClonable {
private final String uniqueName;
private final String className;
- private final MConnectionForms connectionForms;
- private final MJobForms fromJobForms;
- private final MJobForms toJobForms;
- String version;
-
- public MConnector(String uniqueName, String className,
-String version, MConnectionForms connectionForms,
-MJobForms fromJobForms, MJobForms toJobForms) {
+ private final String version;
+ private final MLinkConfig linkConfig;
+ private final MFromConfig fromConfig;
+ private final MToConfig toConfig;
+
+ public MConnector(String uniqueName, String className, String version,
MLinkConfig linkConfig,
+ MFromConfig fromConfig, MToConfig toConfig) {
this.version = version;
-this.connectionForms = connectionForms;
-this.fromJobForms = fromJobForms;
-this.toJobForms = toJobForms;
+this.linkConfig = linkConfig;
+this.fromConfig = fromConfig;
+this.toConfig = toConfig;
+// Why are we abusing NPE?
if (uniqueName == null || className == null) {
throw new NullPointerException();
}
@@ -63,17 +62,15 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
@Override
public String toString() {
-MJobForms fromJobForms = this.getJobForms(Direction.FROM);
-MJobForms toJobForms = this.getJobForms(Direction.TO);
StringBuilder sb = new StringBuilder(connector-);
sb.append(uniqueName).append(:).append(getPersistenceId()).append(:);
sb.append(className);
-sb.append(, ).append(getConnectionForms().toString());
-if (fromJobForms != null) {
- sb.append(, ).append(fromJobForms.toString());
+sb.append(, ).append(getLinkConfig().toString());
+if (getConfig(Direction.FROM) != null) {
+ sb.append(, ).append(getConfig(Direction.FROM).toString());
}
-if (toJobForms != null) {
- sb.append(, ).append(toJobForms.toString());
+if (getConfig(Direction.TO) != null) {
+ sb.append(, ).append(getConfig(Direction.TO).toString());
}
return sb.toString();
}
@@ -94,41 +91,39 @@ public final class MConnector extends MPersistableEntity
implements MClonable {
if (supportedDirections.isDirectionSupported(Direction.FROM)
mcSupportedDirections.isDirectionSupported(Direction.FROM)
-
!getJobForms(Direction.FROM).equals(mc.getJobForms(Direction.FROM))) {
+ !getFromConfig().equals(mc.getFromConfig())) {
return false;
}
-if (supportedDirections.isDirectionSupported(Direction.FROM)
-!= mcSupportedDirections.isDirectionSupported(Direction.FROM)) {
+if (supportedDirections.isDirectionSupported(Direction.FROM) !=
mcSupportedDirections
+.isDirectionSupported(Direction.FROM)) {
return false;
}
if (supportedDirections.isDirectionSupported(Direction.TO)
mcSupportedDirections.isDirectionSupported(Direction.TO)
- !getJobForms(Direction.TO).equals(mc.getJobForms(Direction.TO))) {
+ !getToConfig().equals(mc.getToConfig())) {
return false;
}
-if (supportedDirections.isDirectionSupported(Direction.TO)
-!= mcSupportedDirections.isDirectionSupported(Direction.TO)) {
+if (supportedDirections.isDirectionSupported(Direction.TO) !=
mcSupportedDirections
+.isDirectionSupported(Direction.TO)) {
return false;
}
-return uniqueName.equals(mc.uniqueName)
- className.equals(mc.className)
- version.equals(mc.version)
- connectionForms.equals(mc.getConnectionForms());
+return uniqueName.equals(mc.uniqueName) className.equals(mc.className)
+ version.equals(mc.version)
linkConfig.equals((mc.getLinkConfig()));
}
@Override
public int hashCode() {
SupportedDirections supportedDirections = getSupportedDirections();
-
[47/50] [abbrv] git commit: SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M) (Veena Basavaraj via Abraham Elmahrek) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/8362c73c Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/8362c73c Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/8362c73c Branch: refs/heads/sqoop2 Commit: 8362c73cc01313d2510a659bf296efd727bb1d4e Parents: e540668 Author: Abraham Elmahrek abra...@elmahrek.com Authored: Thu Oct 2 23:29:50 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 18:22:51 2014 -0700 -- .../org/apache/sqoop/client/SqoopClient.java| 119 +-- .../request/ConnectorResourceRequest.java | 4 +- .../request/DriverConfigResourceRequest.java| 15 +- .../client/request/JobResourceRequest.java | 26 +- .../client/request/LinkResourceRequest.java | 10 +- .../client/request/SqoopResourceRequests.java | 4 +- .../apache/sqoop/client/TestSqoopClient.java| 73 +- .../org/apache/sqoop/json/ConnectorBean.java| 129 +-- .../java/org/apache/sqoop/json/DriverBean.java | 90 ++ .../org/apache/sqoop/json/DriverConfigBean.java | 92 -- .../java/org/apache/sqoop/json/JobBean.java | 77 +- .../apache/sqoop/json/JobValidationBean.java| 83 +- .../java/org/apache/sqoop/json/LinkBean.java| 96 +- .../apache/sqoop/json/LinkValidationBean.java | 56 +- .../org/apache/sqoop/json/ThrowableBean.java| 2 +- .../apache/sqoop/json/ValidationResultBean.java | 18 +- .../sqoop/json/util/ConfigSerialization.java| 226 + .../sqoop/json/util/FormSerialization.java | 223 - .../json/util/ResourceBundleSerialization.java | 2 - .../java/org/apache/sqoop/model/Config.java | 34 + .../org/apache/sqoop/model/ConfigClass.java | 45 + .../org/apache/sqoop/model/ConfigUtils.java | 637 + .../apache/sqoop/model/ConfigurationClass.java | 7 +- .../main/java/org/apache/sqoop/model/Form.java | 35 - .../java/org/apache/sqoop/model/FormClass.java | 45 - .../java/org/apache/sqoop/model/FormUtils.java | 634 - .../java/org/apache/sqoop/model/MConfig.java| 117 +++ .../org/apache/sqoop/model/MConfigList.java | 124 +++ .../org/apache/sqoop/model/MConfigType.java | 34 + .../apache/sqoop/model/MConnectionForms.java| 54 -- .../java/org/apache/sqoop/model/MConnector.java | 124 ++- .../java/org/apache/sqoop/model/MDriver.java| 82 ++ .../org/apache/sqoop/model/MDriverConfig.java | 65 +- .../main/java/org/apache/sqoop/model/MForm.java | 117 --- .../java/org/apache/sqoop/model/MFormList.java | 124 --- .../java/org/apache/sqoop/model/MFormType.java | 34 - .../org/apache/sqoop/model/MFromConfig.java | 64 ++ .../main/java/org/apache/sqoop/model/MJob.java | 102 +- .../java/org/apache/sqoop/model/MJobForms.java | 55 -- .../main/java/org/apache/sqoop/model/MLink.java | 72 +- .../org/apache/sqoop/model/MLinkConfig.java | 54 ++ .../java/org/apache/sqoop/model/MToConfig.java | 64 ++ .../java/org/apache/sqoop/model/ModelError.java | 6 +- .../java/org/apache/sqoop/utils/ClassUtils.java | 6 +- .../sqoop/validation/ConfigValidationError.java | 52 + .../validation/ConfigValidationResult.java | 98 ++ .../validation/ConfigValidationRunner.java | 177 .../sqoop/validation/ConfigValidator.java | 228 + .../org/apache/sqoop/validation/Message.java| 2 +- .../org/apache/sqoop/validation/Validation.java | 228 - .../sqoop/validation/ValidationError.java | 52 - .../sqoop/validation/ValidationResult.java | 98 -- .../sqoop/validation/ValidationRunner.java | 176 .../org/apache/sqoop/json/ConfigTestUtil.java | 217 + .../apache/sqoop/json/TestConnectorBean.java| 37 +- .../org/apache/sqoop/json/TestDriverBean.java | 62 ++ .../apache/sqoop/json/TestDriverConfigBean.java | 62 -- .../java/org/apache/sqoop/json/TestJobBean.java | 40 +- .../org/apache/sqoop/json/TestLinkBean.java | 56 +- .../java/org/apache/sqoop/json/TestUtil.java| 153 --- .../apache/sqoop/json/TestValidationBean.java | 83 +- .../sqoop/json/TestValidationResultBean.java| 24 +- .../json/util/TestConfigSerialization.java | 157 .../sqoop/json/util/TestFormSerialization.java | 157 .../org/apache/sqoop/model/TestConfigUtils.java | 290 ++ .../org/apache/sqoop/model/TestFormUtils.java | 295 -- .../sqoop/model/TestMAccountableEntity.java | 9 +- .../org/apache/sqoop/model/TestMConfig.java | 86 ++ .../org/apache/sqoop/model/TestMConfigList.java | 55 ++ .../sqoop/model/TestMConnectionForms.java | 48 - .../org/apache/sqoop/model/TestMConnector.java | 144 +-- .../org/apache/sqoop/model/TestMDriver.java | 40 + .../apache/sqoop/model/TestMDriverConfig.java | 47
[49/50] [abbrv] git commit: SQOOP-1477: Sqoop2: Make link and job name unique identifier
SQOOP-1477: Sqoop2: Make link and job name unique identifier
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/4c964a97
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/4c964a97
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/4c964a97
Branch: refs/heads/sqoop2
Commit: 4c964a97be674ead165844274d48fea9ff69fe65
Parents: c78b75a
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Tue Oct 7 11:38:09 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 18:22:51 2014 -0700
--
.../sqoop/repository/RepositoryError.java | 2 +-
.../derby/DerbyRepositoryHandler.java | 18 +++--
.../repository/derby/DerbySchemaConstants.java | 10 +++
.../repository/derby/DerbySchemaQuery.java | 44 +---
.../sqoop/repository/derby/DerbyTestCase.java | 19 --
.../sqoop/repository/derby/TestInternals.java | 71
.../sqoop/repository/derby/TestJobHandling.java | 23 +--
.../repository/derby/TestLinkHandling.java | 12
8 files changed, 156 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/4c964a97/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
--
diff --git
a/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
b/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
index 54e37d9..0c93b50 100644
--- a/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
+++ b/core/src/main/java/org/apache/sqoop/repository/RepositoryError.java
@@ -122,7 +122,7 @@ public enum RepositoryError implements ErrorCode {
JDBCREPO_0026(Upgrade required but not allowed),
/** Invalid links or jobs when upgrading connector **/
- JDBCREPO_0027(Invalid links or jobs when upgrading connector);
+ JDBCREPO_0027(Invalid links or jobs when upgrading connector)
;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/4c964a97/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
index 39702ca..73d8387 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
@@ -415,6 +415,10 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
// Wait to remove SQB_TYPE (IMPORT/EXPORT) until we update data.
// Data updates depend on knowledge of the type of job.
runQuery(QUERY_UPGRADE_TABLE_SQ_JOB_REMOVE_COLUMN_SQB_TYPE, conn);
+
+ // Add unique constraints on job and links.
+ runQuery(QUERY_UPGRADE_TABLE_SQ_JOB_ADD_UNIQUE_CONSTRAINT_NAME, conn);
+ runQuery(QUERY_UPGRADE_TABLE_SQ_LINK_ADD_UNIQUE_CONSTRAINT_NAME, conn);
}
ResultSet rs = null;
@@ -849,7 +853,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
PreparedStatement stmt = null;
ResultSet rs = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_LINK_CHECK);
+ stmt = conn.prepareStatement(STMT_SELECT_LINK_CHECK_BY_ID);
stmt.setLong(1, id);
rs = stmt.executeQuery();
@@ -1164,9 +1168,9 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
job.getJobConfig(Direction.TO).getConfigs(),
conn);
createInputValues(STMT_INSERT_JOB_INPUT,
-job.getPersistenceId(),
-job.getDriverConfig().getConfigs(),
-conn);
+ job.getPersistenceId(),
+ job.getDriverConfig().getConfigs(),
+ conn);
} catch (SQLException ex) {
logException(ex, job);
@@ -1184,7 +1188,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
PreparedStatement stmt = null;
ResultSet rs = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_JOB_CHECK);
+ stmt = conn.prepareStatement(STMT_SELECT_JOB_CHECK_BY_ID);
stmt.setLong(1, id);
rs = stmt.executeQuery();
@@ -1279,7 +1283,7 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
public MJob findJob(long id, Connection conn) {
PreparedStatement stmt = null;
try {
- stmt = conn.prepareStatement(STMT_SELECT_JOB_SINGLE);
+ stmt =
[39/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
index 5dd7970..39702ca 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbyRepositoryHandler.java
@@ -28,31 +28,38 @@ import java.sql.SQLException;
import java.sql.Statement;
import java.sql.Timestamp;
import java.sql.Types;
-import java.util.*;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashSet;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Set;
-import org.apache.log4j.Logger;
import org.apache.commons.lang.StringUtils;
+import org.apache.log4j.Logger;
import org.apache.sqoop.common.Direction;
import org.apache.sqoop.common.DirectionError;
import org.apache.sqoop.common.SqoopException;
import org.apache.sqoop.connector.ConnectorHandler;
import org.apache.sqoop.connector.ConnectorManagerUtils;
import org.apache.sqoop.model.MBooleanInput;
-import org.apache.sqoop.model.MLink;
-import org.apache.sqoop.model.MConnectionForms;
-import org.apache.sqoop.model.MEnumInput;
-import org.apache.sqoop.model.MIntegerInput;
-import org.apache.sqoop.model.MJob;
-import org.apache.sqoop.model.MJobForms;
+import org.apache.sqoop.model.MConfig;
+import org.apache.sqoop.model.MConfigType;
import org.apache.sqoop.model.MConnector;
-import org.apache.sqoop.model.MForm;
-import org.apache.sqoop.model.MFormType;
+import org.apache.sqoop.model.MDriver;
import org.apache.sqoop.model.MDriverConfig;
+import org.apache.sqoop.model.MEnumInput;
+import org.apache.sqoop.model.MFromConfig;
import org.apache.sqoop.model.MInput;
import org.apache.sqoop.model.MInputType;
+import org.apache.sqoop.model.MIntegerInput;
+import org.apache.sqoop.model.MJob;
+import org.apache.sqoop.model.MLink;
+import org.apache.sqoop.model.MLinkConfig;
import org.apache.sqoop.model.MMapInput;
import org.apache.sqoop.model.MStringInput;
import org.apache.sqoop.model.MSubmission;
+import org.apache.sqoop.model.MToConfig;
import org.apache.sqoop.repository.JdbcRepositoryContext;
import org.apache.sqoop.repository.JdbcRepositoryHandler;
import org.apache.sqoop.submission.SubmissionStatus;
@@ -93,74 +100,69 @@ public class DerbyRepositoryHandler extends
JdbcRepositoryHandler {
mc.getUniqueName());
}
mc.setPersistenceId(getConnectorId(mc, conn));
-insertFormsForConnector(mc, conn);
+insertConfigsForConnector(mc, conn);
}
/**
- * Helper method to insert the forms from the into the
- * repository. The job and connector forms within codemc/code will get
- * updated with the id of the forms when this function returns.
- * @param mDriverConfig The MFramework instance to use to upgrade.
- * @param conn JDBC link to use for updating the forms
+ * Helper method to insert the configs from the into the
+ * repository.
+ * @param mDriver The driver instance to use to upgrade.
+ * @param conn JDBC link to use for updating the configs
*/
- private void insertFormsForFramework(MDriverConfig mDriverConfig, Connection
conn) {
-PreparedStatement baseFormStmt = null;
+ private void insertConfigsForDriver(MDriver mDriver, Connection conn) {
+PreparedStatement baseConfigStmt = null;
PreparedStatement baseInputStmt = null;
try{
- baseFormStmt = conn.prepareStatement(STMT_INSERT_FORM_BASE,
+ baseConfigStmt = conn.prepareStatement(STMT_INSERT_CONFIG_BASE,
Statement.RETURN_GENERATED_KEYS);
baseInputStmt = conn.prepareStatement(STMT_INSERT_INPUT_BASE,
Statement.RETURN_GENERATED_KEYS);
- // Register connector forms
- registerForms(null, null, mDriverConfig.getConnectionForms().getForms(),
-MFormType.CONNECTION.name(), baseFormStmt, baseInputStmt);
-
- // Register job forms
- registerForms(null, null, mDriverConfig.getJobForms().getForms(),
-MFormType.JOB.name(), baseFormStmt, baseInputStmt);
+ // Register the job config type, since driver config is per job
+ registerConfigs(null, null, mDriver.getDriverConfig().getConfigs(),
+MConfigType.JOB.name(), baseConfigStmt, baseInputStmt);
} catch (SQLException ex) {
- throw new SqoopException(DerbyRepoError.DERBYREPO_0014,
mDriverConfig.toString(), ex);
+ throw new SqoopException(DerbyRepoError.DERBYREPO_0014,
mDriver.toString(), ex);
} finally {
- closeStatements(baseFormStmt, baseInputStmt);
+ closeStatements(baseConfigStmt, baseInputStmt);
}
}
/**
-
[44/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/common/src/test/java/org/apache/sqoop/json/ConfigTestUtil.java
--
diff --git a/common/src/test/java/org/apache/sqoop/json/ConfigTestUtil.java
b/common/src/test/java/org/apache/sqoop/json/ConfigTestUtil.java
new file mode 100644
index 000..4372171
--- /dev/null
+++ b/common/src/test/java/org/apache/sqoop/json/ConfigTestUtil.java
@@ -0,0 +1,217 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.json;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.ResourceBundle;
+
+import org.apache.sqoop.model.MConfig;
+import org.apache.sqoop.model.MConnector;
+import org.apache.sqoop.model.MDriver;
+import org.apache.sqoop.model.MDriverConfig;
+import org.apache.sqoop.model.MFromConfig;
+import org.apache.sqoop.model.MInput;
+import org.apache.sqoop.model.MIntegerInput;
+import org.apache.sqoop.model.MJob;
+import org.apache.sqoop.model.MLink;
+import org.apache.sqoop.model.MLinkConfig;
+import org.apache.sqoop.model.MStringInput;
+import org.apache.sqoop.model.MToConfig;
+import org.apache.sqoop.utils.MapResourceBundle;
+
+/**
+ *
+ */
+public class ConfigTestUtil {
+ public static MConnector getConnector(String name) {
+return getConnector(name, true, true);
+ }
+
+ public static MConnector getConnector(String name, boolean from, boolean to)
{
+MFromConfig fromConfig = null;
+MToConfig toConfig = null;
+if (from) {
+ fromConfig = getFromConfig();
+}
+if (to) {
+ toConfig = getToConfig();
+}
+return new MConnector(name, name + .class, 1.0-test,
+ getLinkConfig(), fromConfig, toConfig);
+ }
+
+
+ public static MLink getLink(String name) {
+return new MLink(1, getConnector(name).getLinkConfig());
+ }
+
+ public static MJob getJob(String name) {
+return new MJob(1, 2, 1, 2, getConnector(name).getFromConfig(),
getConnector(name)
+.getToConfig(), getDriverConfig());
+ }
+
+ public static MDriverConfig getDriverConfig() {
+ListMInput? inputs;
+MIntegerInput input;
+MConfig config;
+ListMConfig driverConfigs = new ArrayListMConfig();
+inputs = new ArrayListMInput?();
+
+input = new MIntegerInput(numExtractors, false);
+input.setPersistenceId(1);
+inputs.add(input);
+
+input = new MIntegerInput(numLoaders, false);
+input.setPersistenceId(2);
+inputs.add(input);
+
+config = new MConfig(driver, inputs);
+config.setPersistenceId(10);
+driverConfigs.add(config);
+return new MDriverConfig(driverConfigs);
+ }
+
+ public static MLinkConfig getLinkConfig() {
+ListMInput? inputs;
+MStringInput input;
+MConfig config;
+ListMConfig linkConfig = new ArrayListMConfig();
+inputs = new ArrayListMInput?();
+
+input = new MStringInput(url, false, (short) 10);
+input.setPersistenceId(1);
+inputs.add(input);
+
+input = new MStringInput(username, false, (short) 10);
+input.setPersistenceId(2);
+input.setValue(test);
+inputs.add(input);
+
+input = new MStringInput(password, true, (short) 10);
+input.setPersistenceId(3);
+input.setValue(test);
+inputs.add(input);
+
+config = new MConfig(connection, inputs);
+config.setPersistenceId(10);
+linkConfig.add(config);
+
+return new MLinkConfig(linkConfig);
+ }
+
+ static MFromConfig getFromConfig() {
+ListMInput? inputs;
+MStringInput input;
+MConfig config;
+ListMConfig jobConfigs = new ArrayListMConfig();
+
+inputs = new ArrayListMInput?();
+
+input = new MStringInput(A, false, (short) 10);
+input.setPersistenceId(4);
+inputs.add(input);
+
+input = new MStringInput(B, false, (short) 10);
+input.setPersistenceId(5);
+inputs.add(input);
+
+input = new MStringInput(C, false, (short) 10);
+input.setPersistenceId(6);
+inputs.add(input);
+
+config = new MConfig(Z, inputs);
+config.setPersistenceId(11);
+jobConfigs.add(config);
+
+inputs = new ArrayListMInput?();
+
+input = new MStringInput(D, false, (short) 10);
+
[50/50] [abbrv] git commit: SQOOP-1574: Sqoop2: From/To: Rebase against Sqoop2 branch
SQOOP-1574: Sqoop2: From/To: Rebase against Sqoop2 branch
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/d7e04904
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/d7e04904
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/d7e04904
Branch: refs/heads/sqoop2
Commit: d7e04904300ddef91cb70d65efbc16493b206815
Parents: fc8eb9b
Author: Abraham Elmahrek abra...@elmahrek.com
Authored: Thu Oct 9 19:06:32 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 19:39:41 2014 -0700
--
.../apache/sqoop/tools/tool/JSONConstants.java | 4 +-
.../sqoop/tools/tool/RepositoryDumpTool.java| 15 +-
.../sqoop/tools/tool/RepositoryLoadTool.java| 190 +--
3 files changed, 95 insertions(+), 114 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/d7e04904/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
--
diff --git a/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
b/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
index 288cba3..9bec0d0 100644
--- a/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
+++ b/tools/src/main/java/org/apache/sqoop/tools/tool/JSONConstants.java
@@ -28,9 +28,9 @@ public final class JSONConstants {
public static final String CONNECTOR_NAME = connector-name;
public static final String ALL = all;
public static final String NAME = name;
- public static final String CONNECTION_ID = connection-id;
+ public static final String LINK_ID = link-id;
public static final String JOB_ID = job-id;
- public static final String CONNECTIONS = connections;
+ public static final String LINKS = links;
public static final String JOBS = jobs;
public static final String SUBMISSIONS = submissions;
public static final String METADATA = metadata;
http://git-wip-us.apache.org/repos/asf/sqoop/blob/d7e04904/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
--
diff --git
a/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
b/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
index d41b0d2..f89c546 100644
--- a/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
+++ b/tools/src/main/java/org/apache/sqoop/tools/tool/RepositoryDumpTool.java
@@ -25,25 +25,21 @@ import org.apache.commons.cli.CommandLine;
import org.apache.commons.cli.ParseException;
import org.apache.log4j.Logger;
import org.apache.sqoop.connector.ConnectorManager;
-import org.apache.sqoop.json.ConnectionBean;
import org.apache.sqoop.json.JobBean;
+import org.apache.sqoop.json.LinkBean;
import org.apache.sqoop.json.SubmissionBean;
-import org.apache.sqoop.model.MConnector;
import org.apache.sqoop.repository.Repository;
import org.apache.sqoop.repository.RepositoryManager;
import org.apache.sqoop.tools.ConfiguredTool;
import org.apache.sqoop.common.VersionInfo;
-import static org.apache.sqoop.json.util.FormSerialization.ALL;
+import static org.apache.sqoop.json.util.ConfigSerialization.ALL;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
import java.io.BufferedWriter;
import java.io.FileWriter;
import java.io.IOException;
-import java.util.HashMap;
import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
/**
* Write user-created content of Sqoop repository to JSON formatted file
@@ -106,8 +102,8 @@ public class RepositoryDumpTool extends ConfiguredTool {
JSONObject result = new JSONObject();
LOG.info(Dumping Connections with skipSensitive= +
String.valueOf(skipSensitive));
-ConnectionBean connections = new
ConnectionBean(repository.findConnections());
-result.put(JSONConstants.CONNECTIONS,
addConnectorName(connections.extract(skipSensitive)));
+LinkBean links = new LinkBean(repository.findLinks());
+result.put(JSONConstants.LINKS,
addConnectorName(links.extract(skipSensitive)));
LOG.info(Dumping Jobs with skipSensitive= +
String.valueOf(skipSensitive));
JobBean jobs = new JobBean(repository.findJobs());
@@ -134,7 +130,6 @@ public class RepositoryDumpTool extends ConfiguredTool {
}
private JSONObject addConnectorName(JSONObject json) {
-Repository repository = RepositoryManager.getInstance().getRepository();
ConnectorManager connectorManager = ConnectorManager.getInstance();
JSONArray results = (JSONArray) json.get(ALL);
@@ -144,7 +139,7 @@ public class RepositoryDumpTool extends ConfiguredTool {
while (iterator.hasNext()) {
JSONObject result = iterator.next();
Long connectorId = (Long) result.get(JSONConstants.CONNECTOR_ID);
[38/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
--
diff --git
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
index ad42901..951d9b4 100644
---
a/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
+++
b/repository/repository-derby/src/main/java/org/apache/sqoop/repository/derby/DerbySchemaQuery.java
@@ -48,17 +48,17 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_FORM/strong: Form details.
+ * strongSQ_CONFIG/strong: Config details.
* pre
*+--+
- *| SQ_FORM |
+ *| SQ_CONFIG |
*+--+
- *| SQF_ID: BIGINT PK AUTO-GEN |
- *| SQF_CONNECTOR: BIGINT| FK SQ_CONNECTOR(SQC_ID),NULL for
framework
- *| SQF_DIRECTION: VARCHAR(32) | FROM|TO|NULL
- *| SQF_NAME: VARCHAR(64)|
- *| SQF_TYPE: VARCHAR(32)| CONNECTION|JOB
- *| SQF_INDEX: SMALLINT |
+ *| SQ_CFG_ID: BIGINT PK AUTO-GEN |
+ *| SQ_CFG_OWNER: BIGINT| FK SQ_CFG_OWNER(SQC_ID),NULL for
driver
+ *| SQ_CFG_DIRECTION: VARCHAR(32) | FROM|TO|NULL
+ *| SQ_CFG_NAME: VARCHAR(64)|
+ *| SQ_CFG_TYPE: VARCHAR(32)| LINK|JOB
+ *| SQ_CFG_INDEX: SMALLINT |
*+--+
* /pre
* /p
@@ -70,7 +70,7 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
*++
*| SQI_ID: BIGINT PK AUTO-GEN |
*| SQI_NAME: VARCHAR(64) |
- *| SQI_FORM: BIGINT | FK SQ_FORM(SQF_ID)
+ *| SQI_CONFIG: BIGINT | FK SQ_CONFIG(SQ_CFG_ID)
*| SQI_INDEX: SMALLINT|
*| SQI_TYPE: VARCHAR(32) | STRING|MAP
*| SQI_STRMASK: BOOLEAN |
@@ -80,19 +80,19 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_CONNECTION/strong: Stored connections
+ * strongSQ_LINK/strong: Stored connections
* pre
*++
- *| SQ_CONNECTION |
+ *| SQ_LINK |
*++
- *| SQN_ID: BIGINT PK AUTO-GEN |
- *| SQN_NAME: VARCHAR(64) |
- *| SQN_CONNECTOR: BIGINT | FK SQ_CONNECTOR(SQC_ID)
- *| SQN_CREATION_USER: VARCHAR(32) |
- *| SQN_CREATION_DATE: TIMESTAMP |
- *| SQN_UPDATE_USER: VARCHAR(32) |
- *| SQN_UPDATE_DATE: TIMESTAMP |
- *| SQN_ENABLED: BOOLEAN |
+ *| SQ_LNK_ID: BIGINT PK AUTO-GEN |
+ *| SQ_LNK_NAME: VARCHAR(64) |
+ *| SQ_LNK_CONNECTOR: BIGINT | FK SQ_CONNECTOR(SQC_ID)
+ *| SQ_LNK_CREATION_USER: VARCHAR(32) |
+ *| SQ_LNK_CREATION_DATE: TIMESTAMP |
+ *| SQ_LNK_UPDATE_USER: VARCHAR(32) |
+ *| SQ_LNK_UPDATE_DATE: TIMESTAMP |
+ *| SQ_LNK_ENABLED: BOOLEAN |
*++
* /pre
* /p
@@ -104,8 +104,8 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
*++
*| SQB_ID: BIGINT PK AUTO-GEN |
*| SQB_NAME: VARCHAR(64) |
- *| SQB_FROM_CONNECTION: BIGINT| FK SQ_CONNECTION(SQN_ID)
- *| SQB_TO_CONNECTION: BIGINT | FK SQ_CONNECTION(SQN_ID)
+ *| SQB_FROM_LINK: BIGINT| FK SQ_LINK(SQ_LNK_ID)
+ *| SQB_TO_LINK: BIGINT | FK SQ_LINK(SQ_LNK_ID)
*| SQB_CREATION_USER: VARCHAR(32) |
*| SQB_CREATION_DATE: TIMESTAMP |
*| SQB_UPDATE_USER: VARCHAR(32) |
@@ -115,14 +115,14 @@ import static
org.apache.sqoop.repository.derby.DerbySchemaConstants.*;
* /pre
* /p
* p
- * strongSQ_CONNECTION_INPUT/strong: N:M relationship connection and input
+ * strongSQ_LINK_INPUT/strong: N:M relationship link and input
* pre
*++
- *| SQ_CONNECTION_INPUT|
+ *| SQ_LINK_INPUT|
*++
- *| SQNI_CONNECTION: BIGINT PK | FK SQ_CONNECTION(SQN_ID)
- *| SQNI_INPUT: BIGINT PK | FK SQ_INPUT(SQI_ID)
- *| SQNI_VALUE: LONG VARCHAR |
+ *| SQ_LNKI_LINK: BIGINT PK | FK SQ_LINK(SQ_LNK_ID)
+ *| SQ_LNKI_INPUT: BIGINT PK | FK SQ_INPUT(SQI_ID)
+ *| SQ_LNKI_VALUE: LONG VARCHAR |
*++
* /pre
* /p
@@ -221,17 +221,17 @@ public final class DerbySchemaQuery {
+ COLUMN_SQC_VERSION + VARCHAR(64)
+ );
- // DDL: Create table SQ_FORM
- public static final
[32/50] [abbrv] git commit: SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
SQOOP-1545: Sqoop2: From/To: Add supported directions to Repository
(Abraham Elmahrek via Jarek Jarcec Cecho)
Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/fc74316f
Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/fc74316f
Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/fc74316f
Branch: refs/heads/sqoop2
Commit: fc74316fbdcee80272106064ac4e1e747c300b97
Parents: b600036
Author: Jarek Jarcec Cecho jar...@apache.org
Authored: Thu Oct 9 08:36:35 2014 -0700
Committer: Abraham Elmahrek abra...@elmahrek.com
Committed: Thu Oct 9 18:22:51 2014 -0700
--
.../sqoop/common/SupportedDirections.java | 79 +++-
.../sqoop/common/TestSupportedDirections.java | 19 +
.../sqoop/repository/derby/DerbyRepoError.java | 8 +
.../derby/DerbyRepositoryHandler.java | 398 ---
.../repository/derby/DerbySchemaConstants.java | 53 +++
.../repository/derby/DerbySchemaQuery.java | 164 +++-
.../sqoop/repository/derby/DerbyTestCase.java | 131 --
.../repository/derby/TestConnectorHandling.java | 62 ++-
.../sqoop/repository/derby/TestJobHandling.java | 4 -
.../sqoop/shell/ShowConnectorFunction.java | 30 +-
.../sqoop/connector/spi/SqoopConnector.java | 2 +-
11 files changed, 808 insertions(+), 142 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/sqoop/blob/fc74316f/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
index 25ba276..c527117 100644
--- a/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
+++ b/common/src/main/java/org/apache/sqoop/common/SupportedDirections.java
@@ -20,7 +20,9 @@ package org.apache.sqoop.common;
/**
* Represents which Directions are supported.
*/
-public class SupportedDirections {
+public class SupportedDirections implements ComparableSupportedDirections {
+ private static final char SUPPORTED_DIRECTIONS_SEPARATOR = '/';
+
private boolean from;
private boolean to;
@@ -38,4 +40,79 @@ public class SupportedDirections {
return direction == Direction.FROM from
|| direction == Direction.TO to;
}
+
+ /**
+ * @return String FROM, TO, FROM/TO, .
+ */
+ public String toString() {
+StringBuffer buffer = new StringBuffer();
+
+if (isDirectionSupported(Direction.FROM)) {
+ buffer.append(Direction.FROM);
+
+ if (isDirectionSupported(Direction.TO)) {
+buffer.append(SUPPORTED_DIRECTIONS_SEPARATOR);
+buffer.append(Direction.TO);
+ }
+} else if (isDirectionSupported(Direction.TO)) {
+ buffer.append(Direction.TO);
+}
+
+return buffer.toString();
+ }
+
+ public static SupportedDirections fromString(String supportedDirections) {
+boolean from = false, to = false;
+
+if (supportedDirections != null !supportedDirections.equals()) {
+ for (String direction : supportedDirections.split(/)) {
+switch (Direction.valueOf(direction)) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+ }
+}
+
+return new SupportedDirections(from, to);
+ }
+
+ public static SupportedDirections fromDirection(Direction direction) {
+boolean from = false, to = false;
+switch (direction) {
+ case FROM:
+from = true;
+break;
+
+ case TO:
+to = true;
+break;
+}
+return new SupportedDirections(from, to);
+ }
+
+ @Override
+ public int compareTo(SupportedDirections o) {
+int hash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ hash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ hash |= 2;
+}
+
+int oHash = 0;
+if (this.isDirectionSupported(Direction.FROM)) {
+ oHash |= 1;
+}
+if (this.isDirectionSupported(Direction.TO)) {
+ oHash |= 2;
+}
+
+return hash - oHash;
+ }
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/fc74316f/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
--
diff --git
a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
index 4fbaf82..4f0cdd6 100644
--- a/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
+++ b/common/src/test/java/org/apache/sqoop/common/TestSupportedDirections.java
@@ -52,4 +52,23 @@ public class TestSupportedDirections {
Assert.assertFalse(
[42/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
index 8b23144..12ceb21 100644
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfig.java
@@ -18,7 +18,7 @@
package org.apache.sqoop.connector.jdbc.configuration;
import org.apache.sqoop.connector.jdbc.GenericJdbcConnectorConstants;
-import org.apache.sqoop.model.FormClass;
+import org.apache.sqoop.model.ConfigClass;
import org.apache.sqoop.model.Input;
import org.apache.sqoop.model.Validator;
import org.apache.sqoop.validation.Status;
@@ -28,7 +28,7 @@ import org.apache.sqoop.validation.validators.NullOrContains;
/**
*
*/
-@FormClass( validators = {@Validator(FromJobConfig.FormValidator.class)})
+@ConfigClass( validators = {@Validator(FromJobConfig.ConfigValidator.class)})
public class FromJobConfig {
@Input(size = 50)
public String schemaName;
@@ -51,16 +51,16 @@ public class FromJobConfig {
@Input(size = 50)
public String boundaryQuery;
- public static class FormValidator extends AbstractValidatorFromJobConfig {
+ public static class ConfigValidator extends AbstractValidatorFromJobConfig
{
@Override
-public void validate(FromJobConfig form) {
- if(form.tableName == null form.sql == null) {
+public void validate(FromJobConfig config) {
+ if(config.tableName == null config.sql == null) {
addMessage(Status.UNACCEPTABLE, Either table name or SQL must be
specified);
}
- if(form.tableName != null form.sql != null) {
+ if(config.tableName != null config.sql != null) {
addMessage(Status.UNACCEPTABLE, Both table name and SQL cannot be
specified);
}
- if(form.schemaName != null form.sql != null) {
+ if(config.schemaName != null config.sql != null) {
addMessage(Status.UNACCEPTABLE, Both schema name and SQL cannot be
specified);
}
}
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
index b036421..39e8edd 100644
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/FromJobConfiguration.java
@@ -18,14 +18,14 @@
package org.apache.sqoop.connector.jdbc.configuration;
import org.apache.sqoop.model.ConfigurationClass;
-import org.apache.sqoop.model.Form;
+import org.apache.sqoop.model.Config;
/**
*
*/
@ConfigurationClass
public class FromJobConfiguration {
- @Form public FromJobConfig fromJobConfig;
+ @Config public FromJobConfig fromJobConfig;
public FromJobConfiguration() {
fromJobConfig = new FromJobConfig();
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/LinkConfig.java
--
diff --git
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/LinkConfig.java
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/LinkConfig.java
index 7b76eeb..be86855 100644
---
a/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/LinkConfig.java
+++
b/connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/configuration/LinkConfig.java
@@ -17,7 +17,7 @@
*/
package org.apache.sqoop.connector.jdbc.configuration;
-import org.apache.sqoop.model.FormClass;
+import org.apache.sqoop.model.ConfigClass;
import org.apache.sqoop.model.Input;
import org.apache.sqoop.model.Validator;
import org.apache.sqoop.validation.Status;
@@ -33,7 +33,7 @@ import java.util.Map;
/**
*
*/
-@FormClass(validators = {@Validator(LinkConfig.FormValidator.class)})
+@ConfigClass(validators = {@Validator(LinkConfig.ConfigValidator.class)})
public class LinkConfig {
@Input(size = 128, validators =
[46/50] [abbrv] SQOOP-1498: Sqoop2: Repository Object refactoring (objects prefixed with M)
http://git-wip-us.apache.org/repos/asf/sqoop/blob/8362c73c/common/src/main/java/org/apache/sqoop/json/util/ConfigSerialization.java
--
diff --git
a/common/src/main/java/org/apache/sqoop/json/util/ConfigSerialization.java
b/common/src/main/java/org/apache/sqoop/json/util/ConfigSerialization.java
new file mode 100644
index 000..cec46f6
--- /dev/null
+++ b/common/src/main/java/org/apache/sqoop/json/util/ConfigSerialization.java
@@ -0,0 +1,226 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.sqoop.json.util;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.sqoop.common.SqoopException;
+import org.apache.sqoop.model.MBooleanInput;
+import org.apache.sqoop.model.MEnumInput;
+import org.apache.sqoop.model.MConfig;
+import org.apache.sqoop.model.MConfigType;
+import org.apache.sqoop.model.MInput;
+import org.apache.sqoop.model.MInputType;
+import org.apache.sqoop.model.MIntegerInput;
+import org.apache.sqoop.model.MMapInput;
+import org.apache.sqoop.model.MStringInput;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Convenient static methods for serializing config objects.
+ */
+public final class ConfigSerialization {
+
+ public static final String ALL = all;
+ public static final String ID = id;
+ public static final String NAME = name;
+ public static final String VERSION = version;
+ public static final String CLASS = class;
+ public static final String ENABLED = enabled;
+ public static final String CREATION_USER = creation-user;
+ public static final String CREATION_DATE = creation-date;
+ public static final String UPDATE_USER = update-user;
+ public static final String UPDATE_DATE = update-date;
+ // TODO(VB): Move these constants to connector bean
+ public static final String CONNECTOR_LINK_CONFIG = link-config;
+ public static final String CONNECTOR_JOB_CONFIG = job-config;
+ // TODO:move these configs to driver bean
+ public static final String DRIVER_VERSION = driver-version;
+ public static final String DRIVER_CONFIG = driver-config;
+
+ public static final String CONFIG_NAME = name;
+ public static final String CONFIG_TYPE = type;
+ public static final String CONFIG_INPUTS = inputs;
+ public static final String CONFIG_INPUT_NAME = name;
+ public static final String CONFIG_INPUT_TYPE = type;
+ public static final String CONFIG_INPUT_SENSITIVE = sensitive;
+ public static final String CONFIG_INPUT_SIZE = size;
+ public static final String CONFIG_INPUT_VALUE = value;
+ public static final String CONFIG_INPUT_VALUES = values;
+
+ /**
+ * Transform given list of configs to JSON Array object.
+ *
+ * @param mConfigs List of configs.
+ * @return JSON object with serialized config of the list.
+ */
+ @SuppressWarnings(unchecked)
+ public static JSONArray extractConfigList(ListMConfig mConfigs, boolean
skipSensitive) {
+JSONArray configs = new JSONArray();
+
+for (MConfig mConfig : mConfigs) {
+ configs.add(extractConfig(mConfig, skipSensitive));
+}
+
+return configs;
+ }
+
+ /**
+ * Transform given config to JSON Object.
+ *
+ * @param mConfig Given MConfig instance
+ * @param skipSensitive conditionally add sensitive input values
+ * @return Serialized JSON object.
+ */
+ @SuppressWarnings(unchecked)
+ static JSONObject extractConfig(MConfig mConfig, boolean skipSensitive) {
+JSONObject config = new JSONObject();
+config.put(ID, mConfig.getPersistenceId());
+config.put(CONFIG_NAME, mConfig.getName());
+config.put(CONFIG_TYPE, MConfigType.LINK.toString());
+JSONArray mInputs = new JSONArray();
+config.put(CONFIG_INPUTS, mInputs);
+
+for (MInput? mInput : mConfig.getInputs()) {
+ JSONObject input = new JSONObject();
+ input.put(ID, mInput.getPersistenceId());
+ input.put(CONFIG_INPUT_NAME, mInput.getName());
+ input.put(CONFIG_INPUT_TYPE, mInput.getType().toString());
+ input.put(CONFIG_INPUT_SENSITIVE, mInput.isSensitive());
+
+ // String specific serialization
+ if (mInput.getType() ==
[48/50] [abbrv] git commit: SQOOP-1573: Sqoop2: From/To: Fix driver labels
SQOOP-1573: Sqoop2: From/To: Fix driver labels (Abraham Elmahrek via Jarek Jarcec Cecho) Project: http://git-wip-us.apache.org/repos/asf/sqoop/repo Commit: http://git-wip-us.apache.org/repos/asf/sqoop/commit/fc8eb9b5 Tree: http://git-wip-us.apache.org/repos/asf/sqoop/tree/fc8eb9b5 Diff: http://git-wip-us.apache.org/repos/asf/sqoop/diff/fc8eb9b5 Branch: refs/heads/sqoop2 Commit: fc8eb9b54fef270a9b308ef5e87072a39dd9ff30 Parents: fc74316 Author: Jarek Jarcec Cecho jar...@apache.org Authored: Thu Oct 9 13:59:05 2014 -0700 Committer: Abraham Elmahrek abra...@elmahrek.com Committed: Thu Oct 9 18:22:51 2014 -0700 -- core/src/main/resources/driver-config.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/sqoop/blob/fc8eb9b5/core/src/main/resources/driver-config.properties -- diff --git a/core/src/main/resources/driver-config.properties b/core/src/main/resources/driver-config.properties index 78c3b38..e005775 100644 --- a/core/src/main/resources/driver-config.properties +++ b/core/src/main/resources/driver-config.properties @@ -19,7 +19,7 @@ # Throttling From # -throttthrottlingConfigling.label = Throttling resources +throttlingConfig.label = Throttling resources throttlingConfig.help = Set throttling boundaries to not overload your systems throttlingConfig.numExtractors.label = Extractors
