[jira] [Commented] (SQOOP-2951) Not able to use sqoop command on DB where logging disabled
[ https://issues.apache.org/jira/browse/SQOOP-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710712#comment-16710712 ] Manoj Durisheti commented on SQOOP-2951: Hi, Any update on this or any workaround please? Command and Error: sqoop list-tables --driver com.informix.jdbc.IfxDriver --connect jdbc:informix-sqli://HOST.dev.lcl:50001/abc:informixserver=abc_HOST --username sqoopx --password x 18/12/05 14:30:02 ERROR manager.SqlManager: Error reading database metadata: java.sql.SQLException: No Transaction Isolation on non-logging db's java.sql.SQLException: No Transaction Isolation on non-logging db's at com.informix.util.IfxErrMsg.buildException(IfxErrMsg.java:474) at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:443) at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:394) at com.informix.jdbc.IfxSqliConnect.setTransactionIsolation(IfxSqliConnect.java:3041) at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:910) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:539) at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:225) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.main(Sqoop.java:243) Could not retrieve tables list from server 18/12/05 14:30:02 ERROR tool.ListTablesTool: manager.listTables() returned null > Not able to use sqoop command on DB where logging disabled > -- > > Key: SQOOP-2951 > URL: https://issues.apache.org/jira/browse/SQOOP-2951 > Project: Sqoop > Issue Type: Wish > Components: connectors >Reporter: Ravikumar Ranga >Priority: Major > > How to connect a database which is kind of un-logging (Transaction logging > disabled) from sqoop.With Normal command if I try to connect such DB getting > error. > Syntax of command that been used. > sqoop list-tables --driver com.informix.jdbc.IfxDriver --connect > "jdbc:informix-sqli://XX.XX.XX.22:44XX/pdr:INFORMIXSERVER=zfindvXXX_x" > --username l407 --password > Error > ERROR manager.SqlManager: Error reading database metadata: > java.sql.SQLException: No Transaction Isolation on non-logging db's -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[ https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710229#comment-16710229 ] ASF GitHub Bot commented on SQOOP-3396: --- Github user szvasas commented on a diff in the pull request: https://github.com/apache/sqoop/pull/60#discussion_r239107983 --- Diff: src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesParquetImportTestBase.java --- @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.sqoop.importjob.numerictypes; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; +import org.apache.parquet.schema.MessageType; +import org.apache.parquet.schema.OriginalType; +import org.apache.sqoop.importjob.configuration.ParquetTestConfiguration; +import org.apache.sqoop.testutil.ArgumentArrayBuilder; +import org.apache.sqoop.testutil.NumericTypesTestUtils; +import org.apache.sqoop.util.ParquetReader; +import org.junit.Before; --- End diff -- Unused import > Add parquet numeric support for Parquet in Hive import > -- > > Key: SQOOP-3396 > URL: https://issues.apache.org/jira/browse/SQOOP-3396 > Project: Sqoop > Issue Type: Sub-task >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request: https://github.com/apache/sqoop/pull/60#discussion_r239108068 --- Diff: src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesParquetImportTestBase.java --- @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.sqoop.importjob.numerictypes; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; --- End diff -- Unused import ---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[ https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710230#comment-16710230 ] ASF GitHub Bot commented on SQOOP-3396: --- Github user szvasas commented on a diff in the pull request: https://github.com/apache/sqoop/pull/60#discussion_r239108068 --- Diff: src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesParquetImportTestBase.java --- @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.sqoop.importjob.numerictypes; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; --- End diff -- Unused import > Add parquet numeric support for Parquet in Hive import > -- > > Key: SQOOP-3396 > URL: https://issues.apache.org/jira/browse/SQOOP-3396 > Project: Sqoop > Issue Type: Sub-task >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request: https://github.com/apache/sqoop/pull/60#discussion_r239107983 --- Diff: src/test/org/apache/sqoop/importjob/numerictypes/NumericTypesParquetImportTestBase.java --- @@ -0,0 +1,83 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.sqoop.importjob.numerictypes; + +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.fs.Path; +import org.apache.parquet.schema.MessageType; +import org.apache.parquet.schema.OriginalType; +import org.apache.sqoop.importjob.configuration.ParquetTestConfiguration; +import org.apache.sqoop.testutil.ArgumentArrayBuilder; +import org.apache.sqoop.testutil.NumericTypesTestUtils; +import org.apache.sqoop.util.ParquetReader; +import org.junit.Before; --- End diff -- Unused import ---
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057381
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
+new Object[]{"STRING", Schema.create(Schema.Type.STRING), new
SqoopOptions()},
+new Object[]{"BINARY", Schema.create(Schema.Type.BYTES), new
SqoopOptions()},
--- End diff --
Yeah, make sense
---
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057294
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
--- End diff --
I've added a bunch of static imports (also for Schema.Type) so now the
whole thing is a lot more readable.
---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710047#comment-16710047
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057381
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
+new Object[]{"STRING", Schema.create(Schema.Type.STRING), new
SqoopOptions()},
+new Object[]{"BINARY", Schema.create(Schema.Type.BYTES), new
SqoopOptions()},
--- End diff --
Yeah, make sense
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710044#comment-16710044
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057124
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
--- End diff --
removed
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710046#comment-16710046
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057294
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
--- End diff --
I've added a bunch of static imports (also for Schema.Type) so now the
whole thing is a lot more readable.
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239057124
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
--- End diff --
removed
---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709884#comment-16709884
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239008539
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
--- End diff --
removed
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239008539
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
--- End diff --
removed
---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709883#comment-16709883
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239008265
--- Diff: src/java/org/apache/sqoop/hive/HiveTypes.java ---
@@ -83,27 +89,58 @@ public static String toHiveType(int sqlType) {
}
}
- public static String toHiveType(Schema.Type avroType) {
- switch (avroType) {
-case BOOLEAN:
- return HIVE_TYPE_BOOLEAN;
-case INT:
- return HIVE_TYPE_INT;
-case LONG:
- return HIVE_TYPE_BIGINT;
-case FLOAT:
- return HIVE_TYPE_FLOAT;
-case DOUBLE:
- return HIVE_TYPE_DOUBLE;
-case STRING:
-case ENUM:
- return HIVE_TYPE_STRING;
-case BYTES:
-case FIXED:
- return HIVE_TYPE_BINARY;
-default:
- return null;
+ public static String toHiveType(Schema schema, SqoopOptions options) {
+if (schema.getType() == Schema.Type.UNION) {
+ for (Schema subSchema : schema.getTypes()) {
+if (subSchema.getType() != Schema.Type.NULL) {
+ return toHiveType(subSchema, options);
+}
+ }
+}
+
+Schema.Type avroType = schema.getType();
+switch (avroType) {
+ case BOOLEAN:
+return HIVE_TYPE_BOOLEAN;
+ case INT:
+return HIVE_TYPE_INT;
+ case LONG:
+return HIVE_TYPE_BIGINT;
+ case FLOAT:
+return HIVE_TYPE_FLOAT;
+ case DOUBLE:
+return HIVE_TYPE_DOUBLE;
+ case STRING:
+ case ENUM:
+return HIVE_TYPE_STRING;
+ case BYTES:
+return mapToDecimalOrBinary(schema, options);
+ case FIXED:
+return HIVE_TYPE_BINARY;
+ default:
+throw new RuntimeException(String.format("There is no Hive type
mapping defined for the Avro type of: %s ", avroType.getName()));
+}
+ }
+
+ private static String mapToDecimalOrBinary(Schema schema, SqoopOptions
options) {
+boolean logicalTypesEnabled =
options.getConf().getBoolean(ConfigurationConstants.PROP_ENABLE_PARQUET_LOGICAL_TYPE_DECIMAL,
false);
+if (logicalTypesEnabled && schema.getLogicalType() != null &&
schema.getLogicalType() instanceof Decimal) {
+ Decimal decimal = (Decimal) schema.getLogicalType();
+
+ // trimming precision and scale to Hive's maximum values.
+ int precision = Math.min(HiveDecimal.MAX_PRECISION,
decimal.getPrecision());
+ if (precision < decimal.getPrecision()) {
+LOG.warn("Warning! Precision in the Hive table definition will be
smaller than the actual precision of the column on storage! Hive may not be
able to read data from this column.");
--- End diff --
I created SQOOP-3418 to cover the documentation part.
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (SQOOP-3418) Document decimal support in Hive external import into parquet files
Fero Szabo created SQOOP-3418: - Summary: Document decimal support in Hive external import into parquet files Key: SQOOP-3418 URL: https://issues.apache.org/jira/browse/SQOOP-3418 Project: Sqoop Issue Type: Task Reporter: Fero Szabo Assignee: Fero Szabo Remember to note the limitations in Hive i.e. the max scale and precision is 38 and how it behaves in edge cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user fszabo2 commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r239008265
--- Diff: src/java/org/apache/sqoop/hive/HiveTypes.java ---
@@ -83,27 +89,58 @@ public static String toHiveType(int sqlType) {
}
}
- public static String toHiveType(Schema.Type avroType) {
- switch (avroType) {
-case BOOLEAN:
- return HIVE_TYPE_BOOLEAN;
-case INT:
- return HIVE_TYPE_INT;
-case LONG:
- return HIVE_TYPE_BIGINT;
-case FLOAT:
- return HIVE_TYPE_FLOAT;
-case DOUBLE:
- return HIVE_TYPE_DOUBLE;
-case STRING:
-case ENUM:
- return HIVE_TYPE_STRING;
-case BYTES:
-case FIXED:
- return HIVE_TYPE_BINARY;
-default:
- return null;
+ public static String toHiveType(Schema schema, SqoopOptions options) {
+if (schema.getType() == Schema.Type.UNION) {
+ for (Schema subSchema : schema.getTypes()) {
+if (subSchema.getType() != Schema.Type.NULL) {
+ return toHiveType(subSchema, options);
+}
+ }
+}
+
+Schema.Type avroType = schema.getType();
+switch (avroType) {
+ case BOOLEAN:
+return HIVE_TYPE_BOOLEAN;
+ case INT:
+return HIVE_TYPE_INT;
+ case LONG:
+return HIVE_TYPE_BIGINT;
+ case FLOAT:
+return HIVE_TYPE_FLOAT;
+ case DOUBLE:
+return HIVE_TYPE_DOUBLE;
+ case STRING:
+ case ENUM:
+return HIVE_TYPE_STRING;
+ case BYTES:
+return mapToDecimalOrBinary(schema, options);
+ case FIXED:
+return HIVE_TYPE_BINARY;
+ default:
+throw new RuntimeException(String.format("There is no Hive type
mapping defined for the Avro type of: %s ", avroType.getName()));
+}
+ }
+
+ private static String mapToDecimalOrBinary(Schema schema, SqoopOptions
options) {
+boolean logicalTypesEnabled =
options.getConf().getBoolean(ConfigurationConstants.PROP_ENABLE_PARQUET_LOGICAL_TYPE_DECIMAL,
false);
+if (logicalTypesEnabled && schema.getLogicalType() != null &&
schema.getLogicalType() instanceof Decimal) {
+ Decimal decimal = (Decimal) schema.getLogicalType();
+
+ // trimming precision and scale to Hive's maximum values.
+ int precision = Math.min(HiveDecimal.MAX_PRECISION,
decimal.getPrecision());
+ if (precision < decimal.getPrecision()) {
+LOG.warn("Warning! Precision in the Hive table definition will be
smaller than the actual precision of the column on storage! Hive may not be
able to read data from this column.");
--- End diff --
I created SQOOP-3418 to cover the documentation part.
---
[jira] [Commented] (SQOOP-3416) Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable in the gradle build
[ https://issues.apache.org/jira/browse/SQOOP-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709831#comment-16709831 ] ASF GitHub Bot commented on SQOOP-3416: --- Github user fszabo2 commented on the issue: https://github.com/apache/sqoop/pull/64 This became obsolete > Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable > in the gradle build > --- > > Key: SQOOP-3416 > URL: https://issues.apache.org/jira/browse/SQOOP-3416 > Project: Sqoop > Issue Type: Improvement >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Minor > > Since the sqoopThirdPartyLib doesn't have a default value, if one runs the > Oracle tests, one always have to specify the sqoop.thirdparty.lib.dir system > variable. > With this change, we just have to move the downloaded oracle driver to > /var/lib/dir and avoid some typing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SQOOP-3416) Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable in the gradle build
[ https://issues.apache.org/jira/browse/SQOOP-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709832#comment-16709832 ] ASF GitHub Bot commented on SQOOP-3416: --- Github user fszabo2 closed the pull request at: https://github.com/apache/sqoop/pull/64 > Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable > in the gradle build > --- > > Key: SQOOP-3416 > URL: https://issues.apache.org/jira/browse/SQOOP-3416 > Project: Sqoop > Issue Type: Improvement >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Minor > > Since the sqoopThirdPartyLib doesn't have a default value, if one runs the > Oracle tests, one always have to specify the sqoop.thirdparty.lib.dir system > variable. > With this change, we just have to move the downloaded oracle driver to > /var/lib/dir and avoid some typing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SQOOP-3416) Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable in the gradle build
[ https://issues.apache.org/jira/browse/SQOOP-3416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fero Szabo resolved SQOOP-3416. --- Resolution: Won't Fix This became obsolete, as the variable was deleted. > Give the default value of /var/lib/sqoop to the sqoopThirdPartyLib variable > in the gradle build > --- > > Key: SQOOP-3416 > URL: https://issues.apache.org/jira/browse/SQOOP-3416 > Project: Sqoop > Issue Type: Improvement >Reporter: Fero Szabo >Assignee: Fero Szabo >Priority: Minor > > Since the sqoopThirdPartyLib doesn't have a default value, if one runs the > Oracle tests, one always have to specify the sqoop.thirdparty.lib.dir system > variable. > With this change, we just have to move the downloaded oracle driver to > /var/lib/dir and avoid some typing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] sqoop pull request #64: SQOOP-3416 Give the default value of /var/lib/sqoop ...
Github user fszabo2 closed the pull request at: https://github.com/apache/sqoop/pull/64 ---
[GitHub] sqoop issue #64: SQOOP-3416 Give the default value of /var/lib/sqoop to the ...
Github user fszabo2 commented on the issue: https://github.com/apache/sqoop/pull/64 This became obsolete ---
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984411
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
--- End diff --
Do we need the comment at the end of the line?
---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709822#comment-16709822
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984324
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
--- End diff --
We could static import the create methods from Schema so that would make
these lines shorter.
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984033
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
--- End diff --
I would not add options here, because SqoopOptions does not have a nice
toString() method so we won't see anything meaningful in the test output.
---
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709824#comment-16709824
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984033
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
--- End diff --
I would not add options here, because SqoopOptions does not have a nice
toString() method so we won't see anything meaningful in the test output.
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709821#comment-16709821
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984816
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
+new Object[]{"STRING", Schema.create(Schema.Type.STRING), new
SqoopOptions()},
+new Object[]{"BINARY", Schema.create(Schema.Type.BYTES), new
SqoopOptions()},
--- End diff --
I would add one more test case here:
new Object[]{"BINARY", create(Schema.Type.BYTES),
createSqoopOptionsWithLogicalTypesEnabled()},
to make sure that we test the if condition
org/apache/sqoop/hive/HiveTypes.java:127 fully
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SQOOP-3396) Add parquet numeric support for Parquet in Hive import
[
https://issues.apache.org/jira/browse/SQOOP-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709823#comment-16709823
]
ASF GitHub Bot commented on SQOOP-3396:
---
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984411
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
--- End diff --
Do we need the comment at the end of the line?
> Add parquet numeric support for Parquet in Hive import
> --
>
> Key: SQOOP-3396
> URL: https://issues.apache.org/jira/browse/SQOOP-3396
> Project: Sqoop
> Issue Type: Sub-task
>Reporter: Fero Szabo
>Assignee: Fero Szabo
>Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984816
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
+new Object[]{"INT", Schema.create(Schema.Type.INT), new
SqoopOptions()},
+new Object[]{"BIGINT", Schema.create(Schema.Type.LONG), new
SqoopOptions()},
+new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT), new
SqoopOptions()},
+new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE), new
SqoopOptions()},
+new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>()), new SqoopOptions()}, // Schema.Type.ENUM
+new Object[]{"STRING", Schema.create(Schema.Type.STRING), new
SqoopOptions()},
+new Object[]{"BINARY", Schema.create(Schema.Type.BYTES), new
SqoopOptions()},
--- End diff --
I would add one more test case here:
new Object[]{"BINARY", create(Schema.Type.BYTES),
createSqoopOptionsWithLogicalTypesEnabled()},
to make sure that we test the if condition
org/apache/sqoop/hive/HiveTypes.java:127 fully
---
[GitHub] sqoop pull request #60: SQOOP-3396: Add parquet numeric support for Parquet ...
Github user szvasas commented on a diff in the pull request:
https://github.com/apache/sqoop/pull/60#discussion_r238984324
--- Diff:
src/test/org/apache/sqoop/hive/TestHiveTypesForAvroTypeMapping.java ---
@@ -41,30 +44,49 @@
private final String hiveType;
private final Schema schema;
+ private final SqoopOptions options;
- @Parameters(name = "hiveType = {0}, schema = {1}")
+ @Parameters(name = "hiveType = {0}, schema = {1}, options = {2}")
public static Iterable parameters() {
return Arrays.asList(
-new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN)},
-new Object[]{"INT", Schema.create(Schema.Type.INT)},
-new Object[]{"BIGINT", Schema.create(Schema.Type.LONG)},
-new Object[]{"FLOAT", Schema.create(Schema.Type.FLOAT)},
-new Object[]{"DOUBLE", Schema.create(Schema.Type.DOUBLE)},
-new Object[]{"STRING", Schema.createEnum("ENUM", "doc",
"namespce", new ArrayList<>())}, // Schema.Type.ENUM
-new Object[]{"STRING", Schema.create(Schema.Type.STRING)},
-new Object[]{"BINARY", Schema.create(Schema.Type.BYTES)},
-new Object[]{"BINARY", Schema.createFixed("Fixed", "doc", "space",
1) }
-//, new Object[]{"DECIMAL", Schema.create(Schema.Type.UNION).}
+new Object[]{"BOOLEAN", Schema.create(Schema.Type.BOOLEAN), new
SqoopOptions()},
--- End diff --
We could static import the create methods from Schema so that would make
these lines shorter.
---
[jira] [Commented] (SQOOP-3149) Sqoop incremental import - NULL column updates are not pulled into HBase table
[ https://issues.apache.org/jira/browse/SQOOP-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16709729#comment-16709729 ] Szabolcs Vasas commented on SQOOP-3149: --- Hi [~anjaiahspr], I think you have encountered the issue reported in this JIRA: https://issues.apache.org/jira/browse/SQOOP-3267 It is not yet released upstream but might have included in an HDP release, you could check with your account manager. Regards, Szabolcs > Sqoop incremental import - NULL column updates are not pulled into HBase > table > --- > > Key: SQOOP-3149 > URL: https://issues.apache.org/jira/browse/SQOOP-3149 > Project: Sqoop > Issue Type: Bug > Components: connectors/generic, hbase-integration >Affects Versions: 1.4.6 >Reporter: Jilani Shaik >Priority: Major > Fix For: 1.4.7 > > Attachments: hbase_delete_support_in_incremental_import > > > Sqoop incremental import data from any database to HBase, if source table's > column from a row is updated to NULL, then target HBase table still showing > the previous value for that column. > So if you do a scan on the table for that row, HBase shows the previous > values of the column. > Expected Result: Sqoop incremental import, If NULL columns are there in > source, then HBase need not store that and if it already exists need to > delete that column for a given row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
