> On Sept. 25, 2014, 8:53 p.m., Abraham Elmahrek wrote: > > My sense of things are that we'll probably change things up over time. For > > example, in writing the HBase connector, I'm noticing that user defined > > mappings would be absolutely necessary. So adding some kind of selection > > method for users (which we have a Jira for) make sense. Also, with this > > idea, we have a means of matching in the Driver (renames are happening), > > but the connector still needs to know how to write in-order with respect to > > their own Schema (or lack there of sometimes). For now though, this looks > > good from my perspective. > > > > A couple of style notes... then I think we're good. Also, it seems a > > .DS_Store made it in. Please rm -rf it! > > Gwen Shapira wrote: > Sqoop1 doesn't require any user definition to import to HBase. What would > it take to do the same in Sqoop2?
Sqoop1 takes the names of the columns from the source database and inserts records with the same column names with a user defined column family. The TO connector would have to be aware of the incoming Schema to properly do the same thing. This is definitely something that can be addressed in a follow up Jira. - Abraham ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25180/#review54521 ----------------------------------------------------------- On Sept. 25, 2014, 9:31 p.m., Gwen Shapira wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/25180/ > ----------------------------------------------------------- > > (Updated Sept. 25, 2014, 9:31 p.m.) > > > Review request for Sqoop. > > > Repository: sqoop-SQOOP-1367 > > > Description > ------- > > SQOOP-1378 - Sqoop2: From/To: Refactor schema > > > Diffs > ----- > > common/src/main/java/org/apache/sqoop/job/etl/ExtractorContext.java fd73890 > common/src/main/java/org/apache/sqoop/json/SchemaBean.java 468f7ee > common/src/main/java/org/apache/sqoop/json/SubmissionBean.java 9b1ae74 > common/src/main/java/org/apache/sqoop/json/util/SchemaSerialization.java > f6a9bbf > common/src/main/java/org/apache/sqoop/model/MSubmission.java ca21135 > common/src/main/java/org/apache/sqoop/schema/Schema.java bbebab8 > common/src/main/java/org/apache/sqoop/schema/SchemaError.java 7c8c61e > common/src/main/java/org/apache/sqoop/schema/SchemaMatchOption.java > PRE-CREATION > common/src/main/java/org/apache/sqoop/schema/type/Column.java 30c26a3 > > common/src/test/java/org/apache/sqoop/json/util/TestSchemaSerialization.java > ab5bbd4 > > connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestExtractor.java > 776359a > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnector.java > 70833a0 > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsConnectorError.java > 8a095d2 > > connector/connector-hdfs/src/main/java/org/apache/sqoop/connector/hdfs/HdfsInitializer.java > 923f904 > > connector/connector-hdfs/src/test/java/org/apache/sqoop/connector/hdfs/TestExtractor.java > 7942d59 > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java > df5cb9c > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java > 74b9518 > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/AbstractMatcher.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/LocationMatcher.java > PRE-CREATION > > connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/matcher/NameMatcher.java > PRE-CREATION > > connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestCSVIntermediateDataFormat.java > 8c83a71 > core/src/main/java/org/apache/sqoop/driver/JobManager.java 277c6be > docs/src/site/sphinx/Tools.rst ad72cd1 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java > 2ed06a8 > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java > 6680f60 > > execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java > eea0623 > shell/src/main/java/org/apache/sqoop/shell/utils/SubmissionDisplayer.java > 60acfb6 > shell/src/main/resources/shell-resource.properties b59bd81 > > submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java > fe92ac4 > > Diff: https://reviews.apache.org/r/25180/diff/ > > > Testing > ------- > > Added unit tests and did manual testing: > Hdfs->MySQL > MySQL->Hdfs > MySQL->MySQL (name based matching) > > > Thanks, > > Gwen Shapira > >
