[GitHub] [sqoop] dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import
with create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r283023477
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--drop-and-create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableCreationFailsIfNoCreateOrDropTablePresent()
throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+try {
Review comment:
Of course, that is a much cleaner solution. I did follow the pattern of the
original code.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import
with create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r283023206
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
Review comment:
As I saw into the other test cases I found that code duplication is the
pattern here. I am glad that I was wrong.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282973840
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
Review comment:
This test case currently tests only that --drop-and-create-hcatalog-table
and --hcatalog-external-tabl can present together. For testing the
drop-if-exist behavior there could be two test cases added: pre-create an
external table (e.g. via importing one) and then try to import the same table
1. with --drop-and-create-hcatalog-table --> should succeed
2. without --drop-and-create-hcatalog-table --> should throw an exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282976752
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
Review comment:
This and the next ~15 lines present in all the new test cases. Could you
please extract these into separate methods to eliminate code duplication?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282969069
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +770,29 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
Review comment:
Thanks for adding new test cases!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282973840
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
Review comment:
This test case currently tests only that --drop-and-create-hcatalog-table
and --hcatalog-external-tabl can present together. For testing the
drop-if-exist behavior there could be two test cases added: pre-create an
external table (e.g. via importing one) and try to import it
1. with --drop-and-create-hcatalog-table --> should succeed
2. without --drop-and-create-hcatalog-table --> should throw an exception
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w… URL: https://github.com/apache/sqoop/pull/80#discussion_r282968802 ## File path: src/docs/user/hcatalog.txt ## @@ -61,6 +61,10 @@ The presence of the +--hcatalog-table+ option signifies that the import or export job is done using HCatalog tables, and it is a required option for HCatalog jobs. ++--hcatalog-external-table+:: Review comment: Thanks for extending the documentation! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282975080
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +771,80 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableDropAndCreation() throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--drop-and-create-hcatalog-table");
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+runHCatImport(addlArgsArray, TOTAL_RECORDS, table, cols,
+null, true, false);
+ }
+
+ @Test
+ public void testExternalTableCreationFailsIfNoCreateOrDropTablePresent()
throws Exception {
+final int TOTAL_RECORDS = 1 * 10;
+String table = getTableName().toUpperCase();
+ColumnGenerator[] cols = new ColumnGenerator[] {
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(0),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("1", 20), "1", KeyType.STATIC_KEY),
+HCatalogTestUtils.colGenerator(HCatalogTestUtils.forIdx(1),
+"varchar(20)", Types.VARCHAR, HCatFieldSchema.Type.STRING,
0, 0,
+new HiveVarchar("2", 20), "2", KeyType.DYNAMIC_KEY),
+};
+List addlArgsArray = new ArrayList();
+addlArgsArray.add("--hcatalog-external-table");
+addlArgsArray.add("--hcatalog-storage-stanza");
+addlArgsArray.add("\"stored as orc tblproperties
(\"transactional\"=\"false\")\"");
+setExtraArgs(addlArgsArray);
+utils.dropHCatTableIfExists(table, SqoopHCatUtilities.DEFHCATDB);
+try {
Review comment:
Could you please use ExpectedException rule instead of using a try-catch
block here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (SQOOP-3438) Sqoop Import with create hcatalog table for ORC will not work with Hive3 as the table created would be a ACID table and transactional
[ https://issues.apache.org/jira/browse/SQOOP-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837237#comment-16837237 ] Denes Bodo commented on SQOOP-3438: --- I do not know why I created this as a BUG. Now type is changed to improvement. > Sqoop Import with create hcatalog table for ORC will not work with Hive3 as > the table created would be a ACID table and transactional > - > > Key: SQOOP-3438 > URL: https://issues.apache.org/jira/browse/SQOOP-3438 > Project: Sqoop > Issue Type: Improvement > Components: hive-integration >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Time Spent: 1.5h > Remaining Estimate: 0h > > PROBLEM: Running a sqoop import command with the option > --create-hcatalog-table will not work due to the following reasons > When create-hcatalog-table is used it creates the table as a Managed ACID > table. > HCatalog does not support transactional or bucketing table > So customer who need to create a ORC based table cannot use sqoop to create a > ORC based table which means their existing code where if in case they use > sqoop to create these tables would fail. > The current workaround is a two step process > 1. Create the ORC table in hive with the keyword external and set > transactional to false > 2. Then use the sqoop command to load the data into the orc table. > The request is to add in an extra argument in the sqoop command line to > specify that the table is external (example: --hcatalog-external-table )so we > can use the option --hcatalog-storage-stanza "stored as orc tblproperties > (\"transactional\"=\"false\")". > > Thank you [~mbalakrishnan] for your findings. This ticket is created based on > your work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [sqoop] dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import
with create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282867073
##
File path: src/java/org/apache/sqoop/SqoopOptions.java
##
@@ -1654,6 +1656,14 @@ public String getHCatTableName() {
return this.hCatTableName;
}
+ public void useExternalHCatTable(boolean value) {
Review comment:
Thanks @dvoros
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import
with create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282866850
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +770,29 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
Review comment:
One test case for drop and create and one negative is added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dionusos commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w… URL: https://github.com/apache/sqoop/pull/80#discussion_r282866612 ## File path: src/docs/user/hcatalog.txt ## @@ -61,6 +61,10 @@ The presence of the +--hcatalog-table+ option signifies that the import or export job is done using HCatalog tables, and it is a required option for HCatalog jobs. ++--hcatalog-external-table+:: Review comment: You are correct, thanks for notifying me about the use case with delete. I think the provided example fits there. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (SQOOP-3438) Sqoop Import with create hcatalog table for ORC will not work with Hive3 as the table created would be a ACID table and transactional
[ https://issues.apache.org/jira/browse/SQOOP-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denes Bodo updated SQOOP-3438: -- Issue Type: Improvement (was: Bug) > Sqoop Import with create hcatalog table for ORC will not work with Hive3 as > the table created would be a ACID table and transactional > - > > Key: SQOOP-3438 > URL: https://issues.apache.org/jira/browse/SQOOP-3438 > Project: Sqoop > Issue Type: Improvement > Components: hive-integration >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Time Spent: 1h > Remaining Estimate: 0h > > PROBLEM: Running a sqoop import command with the option > --create-hcatalog-table will not work due to the following reasons > When create-hcatalog-table is used it creates the table as a Managed ACID > table. > HCatalog does not support transactional or bucketing table > So customer who need to create a ORC based table cannot use sqoop to create a > ORC based table which means their existing code where if in case they use > sqoop to create these tables would fail. > The current workaround is a two step process > 1. Create the ORC table in hive with the keyword external and set > transactional to false > 2. Then use the sqoop command to load the data into the orc table. > The request is to add in an extra argument in the sqoop command line to > specify that the table is external (example: --hcatalog-external-table )so we > can use the option --hcatalog-storage-stanza "stored as orc tblproperties > (\"transactional\"=\"false\")". > > Thank you [~mbalakrishnan] for your findings. This ticket is created based on > your work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3438) Sqoop Import with create hcatalog table for ORC will not work with Hive3 as the table created would be a ACID table and transactional
[ https://issues.apache.org/jira/browse/SQOOP-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837050#comment-16837050 ] Boglarka Egyed commented on SQOOP-3438: --- [~dionusos] I have reviewed your change, please find it at your pull request. I think this change is more of an Improvement than a Bug, would you mind to change the type of this issue? > Sqoop Import with create hcatalog table for ORC will not work with Hive3 as > the table created would be a ACID table and transactional > - > > Key: SQOOP-3438 > URL: https://issues.apache.org/jira/browse/SQOOP-3438 > Project: Sqoop > Issue Type: Bug > Components: hive-integration >Affects Versions: 1.4.7 >Reporter: Denes Bodo >Assignee: Denes Bodo >Priority: Critical > Time Spent: 1h > Remaining Estimate: 0h > > PROBLEM: Running a sqoop import command with the option > --create-hcatalog-table will not work due to the following reasons > When create-hcatalog-table is used it creates the table as a Managed ACID > table. > HCatalog does not support transactional or bucketing table > So customer who need to create a ORC based table cannot use sqoop to create a > ORC based table which means their existing code where if in case they use > sqoop to create these tables would fail. > The current workaround is a two step process > 1. Create the ORC table in hive with the keyword external and set > transactional to false > 2. Then use the sqoop command to load the data into the orc table. > The request is to add in an extra argument in the sqoop command line to > specify that the table is external (example: --hcatalog-external-table )so we > can use the option --hcatalog-storage-stanza "stored as orc tblproperties > (\"transactional\"=\"false\")". > > Thank you [~mbalakrishnan] for your findings. This ticket is created based on > your work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w… URL: https://github.com/apache/sqoop/pull/80#discussion_r282794994 ## File path: src/docs/user/hcatalog.txt ## @@ -61,6 +61,10 @@ The presence of the +--hcatalog-table+ option signifies that the import or export job is done using HCatalog tables, and it is a required option for HCatalog jobs. ++--hcatalog-external-table+:: Review comment: Using this option makes sense only together with --create-hcatalog-table thus could you please add fail fast scenarios to prevent separate usage? Also could you please update the documentation accordingly? For reference please see SQOOP-816 and SQOOP-3324 where external Hive table option has been introduced and documented. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w… URL: https://github.com/apache/sqoop/pull/80#discussion_r282789524 ## File path: src/docs/user/hcatalog.txt ## @@ -61,6 +61,10 @@ The presence of the +--hcatalog-table+ option signifies that the import or export job is done using HCatalog tables, and it is a required option for HCatalog jobs. ++--hcatalog-external-table+:: Review comment: Could you please extend the documentation of --drop-and-create-hcatalog-table option that it will have affect on external HCat tables too? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [sqoop] ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
ebogi commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282789992
##
File path: src/test/org/apache/sqoop/hcat/HCatalogImportTest.java
##
@@ -770,6 +770,29 @@ public void testTableCreation() throws Exception {
null, true, false);
}
+ @Test
+ public void testExternalTableCreation() throws Exception {
Review comment:
Could you please add more test cases? E.g. for when the table already exists
and either exception is thrown or we drop it based on the usage of
--drop-and-create-hcatalog-table option?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [sqoop] dvoros commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dvoros commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w… URL: https://github.com/apache/sqoop/pull/80#discussion_r282785682 ## File path: src/docs/user/hcatalog.txt ## @@ -61,6 +61,10 @@ The presence of the +--hcatalog-table+ option signifies that the import or export job is done using HCatalog tables, and it is a required option for HCatalog jobs. ++--hcatalog-external-table+:: +Set this flag if you need to create external Hive table for example to store Review comment: Instead of ORC, I'd mention non-transactional tables, since that's stopping you from creating HCat table without this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [sqoop] dvoros commented on a change in pull request #80: SQOOP-3438 Sqoop Import with create hcatalog table for ORC will not w…
dvoros commented on a change in pull request #80: SQOOP-3438 Sqoop Import with
create hcatalog table for ORC will not w…
URL: https://github.com/apache/sqoop/pull/80#discussion_r282785214
##
File path: src/java/org/apache/sqoop/SqoopOptions.java
##
@@ -1654,6 +1656,14 @@ public String getHCatTableName() {
return this.hCatTableName;
}
+ public void useExternalHCatTable(boolean value) {
Review comment:
Nit: Other boolean setter methods are called `set*` here.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
