This is an automatically generated e-mail. To reply, visit:

Review request for Sqoop.

Bugs: SQOOP-3311

Repository: sqoop-trunk


Hive 3 will introduce a switch (HIVE-18294) to create eligible tables as ACID 
by default. This will probably result in increased usage of ACID tables and the 
need to support importing into ACID tables with Sqoop.

Currently the only table format supporting full ACID tables is ORC.

The easiest and most effective way to support importing into these tables would 
be to write out files as ORC and keep using LOAD DATA as we do for all other 
Hive tables (supported since HIVE-17361).

Workaround could be to create table as textfile (as before) and then CTAS from 
that. This would push the responsibility of creating ORC format to Hive. 
However it would result in writing every record twice; in text format and in 

Note that ORC is only necessary for full ACID tables. Insert-only (aka. 
micromanaged) ACID tables can use arbitrary file format.

Supporting full ACID tables would also be the first step in making 
"lastmodified" incremental imports work with Hive.


  ivy.xml 6be4fa2 
  ivy/libraries.properties c44b50b 
  src/java/org/apache/sqoop/SqoopOptions.java 651cebd 
  src/java/org/apache/sqoop/hive/TableDefWriter.java b7a25b7 
  src/java/org/apache/sqoop/mapreduce/DataDrivenImportJob.java a5962ba 
  src/java/org/apache/sqoop/mapreduce/OrcImportMapper.java PRE-CREATION 
  src/java/org/apache/sqoop/tool/BaseSqoopTool.java b02e4fe 
  src/java/org/apache/sqoop/tool/ExportTool.java 060f2c0 
  src/java/org/apache/sqoop/tool/ImportTool.java e992005 
  src/java/org/apache/sqoop/util/OrcUtil.java PRE-CREATION 
  src/test/org/apache/sqoop/TestOrcImport.java PRE-CREATION 
  src/test/org/apache/sqoop/hive/TestTableDefWriter.java 8bdc3be 
  src/test/org/apache/sqoop/orm/TestClassWriter.java 0cc07cf 
  src/test/org/apache/sqoop/util/TestOrcUtil.java PRE-CREATION 

Diff: https://reviews.apache.org/r/66548/diff/1/


- added some unit tests
- tested basic Hive import scenarios on a cluster


daniel voros

Reply via email to