----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25180/#review52664 -----------------------------------------------------------
The biggest design concern I see is which component owns matching schemas (if any). The review posted provides a framework centered implementation: 1. From/To schemas exist. Name and location matching is possible. 2. From schema exists. Location matching only. 3. To schema exists. Location matching only. 4. No schemas exist. Location matching only. The concept of a "Matcher" seems a lot like a mini-transformation layer. Given we haven't actually implemented a transformation layer, it seems like we should delegate more to the Connector. After all, the IntermediateDataFormat is intended to be a common language that all connectors speak. I think the connectors can assume that they will always receive data in order. I do think that a lot of the code implemented here can be reused if transformation layers are added. I'm not 100% on this, so lets hear from more members of the community? common/src/main/java/org/apache/sqoop/model/MSubmission.java <https://reviews.apache.org/r/25180/#comment91557> Required now common/src/main/java/org/apache/sqoop/schema/Schema.java <https://reviews.apache.org/r/25180/#comment91558> import wildcard. connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java <https://reviews.apache.org/r/25180/#comment91589> Switch statement? - Abraham Elmahrek On Sept. 3, 2014, 10:03 p.m., Gwen Shapira wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25180/ > ----------------------------------------------------------- > > (Updated Sept. 3, 2014, 10:03 p.m.) > > > Review request for Sqoop. > > > Repository: sqoop-SQOOP-1367 > > > Description > ------- > > SQOOP-1378 - Sqoop2: From/To: Refactor schema > > > Diffs > ----- > > common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java af03f0a > common/src/main/java/org/apache/sqoop/json/SchemaBean.java 468f7ee > common/src/main/java/org/apache/sqoop/json/SubmissionBean.java 61d6576 > common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java > f6a9bbf > common/src/main/java/org/apache/sqoop/model/MSubmission.java 1edd6ee > common/src/main/java/org/apache/sqoop/schema/Schema.java bbebab8 > common/src/main/java/org/apache/sqoop/schema/SchemaError.java 7c8c61e > common/src/main/java/org/apache/sqoop/schema/SchemaMatchOption.java > PRE-CREATION > common/src/main/java/org/apache/sqoop/schema/type/Column.java 30c26a3 > common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java d87655e > > common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java > ab5bbd4 > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcFromInitializer.java > 87e0862 > > connector/connector-generic-jdbc/src/main/java/org/apache/sqoop/connector/jdbc/GenericJdbcToInitializer.java > 73a49b1 > > connector/connector-generic-jdbc/src/main/resources/generic-jdbc-connector-resources.properties > a1302c0 > > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExtractor.java > 2b1dec2 > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnector.java > 883636c > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnectorError.java > 8a095d2 > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsInitializer.java > d2d12a8 > > connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestExtractor.java > 6ed4087 > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java > 1e8ab52 > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java > 91b594e > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/AbstractMatcher.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/LocationMatcher.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/NameMatcher.java > PRE-CREATION > > connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormatTest.java > df6d30f > core/src/main/java/org/apache/sqoop/framework/JobManager.java b1b37f6 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java > 92414d8 > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java > 1d60ba3 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java > e457cff > shell/src/main/java/org/apache/sqoop/shell/core/Constants.java efabc46 > shell/src/main/java/org/apache/sqoop/shell/utils/SubmissionDisplayer.java > 6dbd870 > shell/src/main/resources/shell-resource.properties 73a19e8 > > submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java > bfa6958 > > Diff: https://reviews.apache.org/r/25180/diff/ > > > Testing > ------- > > Added unit tests and did manual testing: > Hdfs->MySQL > MySQL->Hdfs > MySQL->MySQL (name based matching) > > > Thanks, > > Gwen Shapira > >
