-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23278/
-----------------------------------------------------------
(Updated July 10, 2014, 8:01 a.m.)
Review request for Sqoop.
Changes
-------
Added documentation. Also changed the abstract method getColumnInfo to a
concrete method that throws an exception to allow managers that explicitly
extend from ConnManager
Bugs: SQOOP-1322, SQOOP-1323, SQOOP-1324, SQOOP-1325, SQOOP-1326, and SQOOP-1363
https://issues.apache.org/jira/browse/SQOOP-1322
https://issues.apache.org/jira/browse/SQOOP-1323
https://issues.apache.org/jira/browse/SQOOP-1324
https://issues.apache.org/jira/browse/SQOOP-1325
https://issues.apache.org/jira/browse/SQOOP-1326
https://issues.apache.org/jira/browse/SQOOP-1363
Repository: sqoop-trunk
Description
-------
Consolidated patch for HCat enhancements introduced in Hive 0.13. With Hive
0.13, HCatalog has restored datatype partity with Hive.
Furthermore, HCatalog APIs that Sqoop is currently using (pre HCatalog 0.11
API) were deprecated in Hive 0.12 and have been removed in Hive 0.14 (trunk
version currently in progress). The old HCatalog APIs even in Hive 0.12 and
Hive 0.13 did not include new Hive datatypes like DATE, TIMESTAMP, CHAR,
VARCHAR and DECIMAL.
So, to enhance HCatalog support in Sqoop to support all Hive datatypes we have
to use the new HCatalog APIs that are in the org.apache.hive.hcatalog package.
Note that this means that we will no longer be able to use Hive 0.12 or
earlier to build Sqoop. We need Hive 0.13 to build Sqoop. But Sqoop build
with Hive 0.13 can still work with older Hive versions using the HCatalog
interface if the Hive home used by Sqoop points to hive 0.13 version (the HCAT
api is backward compatible)
Along with support for all Hive datatypes, some of the customer and user
required features have been added - multiple static partition key support and
escape the object names used.
Sorry for a rather large path to review. I split to subtasks but it seemed
more easier to provide a single patch because of the inter-dependencies.
Will create a separate JIRA issue to track documentation updates
Diffs (updated)
-----
build.xml ec5d2fa
ivy.xml 65ef089
src/docs/user/hcatalog.txt b1a3982
src/java/org/apache/sqoop/SqoopOptions.java f1b8b13
src/java/org/apache/sqoop/manager/ConnManager.java 773d246
src/java/org/apache/sqoop/manager/SqlManager.java 58fea05
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableHCatExportMapper.java
a139090
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableHCatImportMapper.java
6f163e9
src/java/org/apache/sqoop/mapreduce/db/netezza/NetezzaExternalTableTextImportMapper.java
acc4a2a
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportFormat.java 47febf7
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportHelper.java e48f6d6
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatExportMapper.java c7e9b8e
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportHelper.java e9606ad
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatImportMapper.java 2d4830a
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatInputSplit.java 5a2e48a
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatRecordReader.java 55604f7
src/java/org/apache/sqoop/mapreduce/hcat/SqoopHCatUtilities.java 25a39be
src/java/org/apache/sqoop/tool/BaseSqoopTool.java ceda9f3
src/java/org/apache/sqoop/tool/CodeGenTool.java c1ea881
src/java/org/apache/sqoop/tool/ExportTool.java 4c7d00c
src/java/org/apache/sqoop/tool/ImportTool.java 6cbb873
src/test/com/cloudera/sqoop/TestConnFactory.java c0b295e
src/test/org/apache/sqoop/hcat/HCatalogExportTest.java 4031973
src/test/org/apache/sqoop/hcat/HCatalogImportTest.java ab08013
src/test/org/apache/sqoop/hcat/HCatalogTestUtils.java abb809f
src/test/org/apache/sqoop/hcat/TestHCatalogBasic.java 45646dd
src/test/org/apache/sqoop/manager/netezza/DirectNetezzaHCatExportManualTest.java
2183b1a
src/test/org/apache/sqoop/manager/netezza/DirectNetezzaHCatImportManualTest.java
36bc53c
Diff: https://reviews.apache.org/r/23278/diff/
Testing
-------
Added new tests for the features being added. All tests and unit tests pass
Ran checkstyle and made sure no new regressions are there except for one issue
with 7 parameters to a method to hcat test class.
Please note that to test HCatalog import and export tests, we have to
explicitly invoke the test classes using command line like below
ant -Dhadoopversion=100 clean test -Dtestcase=HCatalogImportTest
ant -Dhadoopversion=100 clean test -Dtestcase=HCatalogExportTest
Please note that to test with Hadoop 2 Sqoop profiles you have to explictly
build Hive with Hadoop 2 and use the resultant artifacts. I have updated
SQOOP-1064 to reflect that we are now blocked by HIVE-7349 for moving the
HCatalog tests to be run as part of unit tests.
Thanks,
Venkat Ranganathan
