[jira] [Created] (SQOOP-3153) Sqoop export with spec file formats error message could be more verbose
Boglarka Egyed created SQOOP-3153:
-
Summary: Sqoop export with spec file formats error message could
be more verbose
Key: SQOOP-3153
URL: https://issues.apache.org/jira/browse/SQOOP-3153
Project: Sqoop
Issue Type: Improvement
Affects Versions: 1.4.6
Reporter: Boglarka Egyed
Priority: Minor
Currently the error message during an export with --as-sequencefile or
--as-avrodatafile or --as-parquetfile is the following:
{code}
ERROR tool.BaseSqoopTool: Error parsing arguments for export:
ERROR tool.BaseSqoopTool: Unrecognized argument: --as-avrodatafile
{code}
Based on the result of SQOOP-3151 this error message could be more verbose as
it is a bit misleading now.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SQOOP-3153) Sqoop export with --as- error message could be more verbose
[
https://issues.apache.org/jira/browse/SQOOP-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed updated SQOOP-3153:
--
Summary: Sqoop export with --as- error message could be
more verbose (was: Sqoop export with spec file formats error message could be
more verbose)
> Sqoop export with --as- error message could be more verbose
> -
>
> Key: SQOOP-3153
> URL: https://issues.apache.org/jira/browse/SQOOP-3153
> Project: Sqoop
> Issue Type: Improvement
>Affects Versions: 1.4.6
>Reporter: Boglarka Egyed
>Priority: Minor
>
> Currently the error message during an export with --as-sequencefile or
> --as-avrodatafile or --as-parquetfile is the following:
> {code}
> ERROR tool.BaseSqoopTool: Error parsing arguments for export:
> ERROR tool.BaseSqoopTool: Unrecognized argument: --as-avrodatafile
> {code}
> Based on the result of SQOOP-3151 this error message could be more verbose as
> it is a bit misleading now.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (SQOOP-2272) Import decimal columns from mysql to hive 0.14
[ https://issues.apache.org/jira/browse/SQOOP-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lin updated SQOOP-2272: Attachment: SQOOP-2272.patch Allowing sqoop to convert DECIMAL to DECIMAL in Hive > Import decimal columns from mysql to hive 0.14 > -- > > Key: SQOOP-2272 > URL: https://issues.apache.org/jira/browse/SQOOP-2272 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.4.5 >Reporter: Pawan Pawar >Assignee: Eric Lin > Attachments: SQOOP-2272.patch > > > I am importing data from mysql to hive. several columns in source table are > of type decimal. but sqoop convert this types into the double. How can I > import that table with same precision and scale in hive. > My query is: > sqoop import --connect > jdbc:mysql://localhost:3306/SourceDataBase?zeroDateTimeBehavior=convertToNull > --username root --password root --hive-table MyHiveDatabaseName.MyTableName > --hive-import --hive-table MyHiveDatabaseName.MyTableName --query 'select * > from MyTableName where $CONDITIONS' -m 1 --target-dir > /user/hive/warehouse/MyHiveDatabaseName/MyTableName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
Review Request 57551: SQOOP-2272 - Import decimal columns from mysql to hive 0.14
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/57551/ --- Review request for Sqoop, Attila Szabo and Szabolcs Vasas. Repository: sqoop-trunk Description --- Currently Sqoop converts DECIMAL from RDMS into DOUBLE in Hive, which is not correct as user will lose precisions. Since Hive supports DECIMAL long ago, we should support DECIMAL to DECIMAL conversion from Sqoop to Hive. Diffs - src/java/org/apache/sqoop/hive/HiveTypes.java ad00535 src/java/org/apache/sqoop/hive/TableDefWriter.java 32fcca3 src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 4db629f Diff: https://reviews.apache.org/r/57551/diff/1/ Testing --- Test case + maunaul test Thanks, Eric Lin
[jira] [Commented] (SQOOP-2272) Import decimal columns from mysql to hive 0.14
[ https://issues.apache.org/jira/browse/SQOOP-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15907106#comment-15907106 ] Eric Lin commented on SQOOP-2272: - Review: https://reviews.apache.org/r/57551/ > Import decimal columns from mysql to hive 0.14 > -- > > Key: SQOOP-2272 > URL: https://issues.apache.org/jira/browse/SQOOP-2272 > Project: Sqoop > Issue Type: Bug > Components: sqoop2-shell >Affects Versions: 1.4.5 >Reporter: Pawan Pawar >Assignee: Eric Lin > Attachments: SQOOP-2272.patch > > > I am importing data from mysql to hive. several columns in source table are > of type decimal. but sqoop convert this types into the double. How can I > import that table with same precision and scale in hive. > My query is: > sqoop import --connect > jdbc:mysql://localhost:3306/SourceDataBase?zeroDateTimeBehavior=convertToNull > --username root --password root --hive-table MyHiveDatabaseName.MyTableName > --hive-import --hive-table MyHiveDatabaseName.MyTableName --query 'select * > from MyTableName where $CONDITIONS' -m 1 --target-dir > /user/hive/warehouse/MyHiveDatabaseName/MyTableName -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
Eric Lin created SQOOP-3152:
---
Summary: --map-column-hive to support DECIMAL(xx,xx)
Key: SQOOP-3152
URL: https://issues.apache.org/jira/browse/SQOOP-3152
Project: Sqoop
Issue Type: Bug
Reporter: Eric Lin
Assignee: Eric Lin
Priority: Minor
The following command:
sqoop import --connect jdbc:mysql://localhost/test --username root --password
'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver --verbose
--hive-import --hive-database default --hive-table decimal_table
--hive-overwrite --map-column-hive a='DECIMAL(10,4)'
will fail with below error:
{code}
17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should
be the form key=value[,key=value]*
java.lang.IllegalArgumentException: Malformed mapping. Column mapping should
be the form key=value[,key=value]*
at
org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333)
at
org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349)
at
org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Malformed mapping. Column mapping should be the form key=value[,key=value]*
{code}
--map-column-hive should support DECIMAL(10,5) format.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Review Request 57576: SQOOP-3152 - --map-column-hive to support DECIMAL(xx, xx)
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57576/
---
Review request for Sqoop, Attila Szabo and Szabolcs Vasas.
Repository: sqoop-trunk
Description
---
The following command:
sqoop import --connect jdbc:mysql://localhost/test --username root --password
'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver --verbose
--hive-import --hive-database default --hive-table decimal_table
--hive-overwrite --map-column-hive a='DECIMAL(10,4)'
will fail with below error:
{code}
17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping should
be the form key=value[,key=value]*
java.lang.IllegalArgumentException: Malformed mapping. Column mapping should
be the form key=value[,key=value]*
at
org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333)
at
org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349)
at
org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Malformed mapping. Column mapping should be the form key=value[,key=value]*
{code}
--map-column-hive should support DECIMAL(10,5) format.
Diffs
-
src/java/org/apache/sqoop/SqoopOptions.java b33b54b
src/test/com/cloudera/sqoop/hive/TestTableDefWriter.java 4db629f
Diff: https://reviews.apache.org/r/57576/diff/1/
Testing
---
Added test case + manual testing
Thanks,
Eric Lin
[jira] [Commented] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[
https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923261#comment-15923261
]
Eric Lin commented on SQOOP-3152:
-
Review link: https://reviews.apache.org/r/57576/
> --map-column-hive to support DECIMAL(xx,xx)
> ---
>
> Key: SQOOP-3152
> URL: https://issues.apache.org/jira/browse/SQOOP-3152
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: SQOOP-3152.patch
>
>
> The following command:
> sqoop import --connect jdbc:mysql://localhost/test --username root --password
> 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver
> --verbose --hive-import --hive-database default --hive-table decimal_table
> --hive-overwrite --map-column-hive a='DECIMAL(10,4)'
> will fail with below error:
> {code}
> 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping
> should be the form key=value[,key=value]*
> java.lang.IllegalArgumentException: Malformed mapping. Column mapping should
> be the form key=value[,key=value]*
> at
> org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333)
> at
> org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349)
> at
> org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198)
> at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011)
> at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:135)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> Malformed mapping. Column mapping should be the form key=value[,key=value]*
> {code}
> --map-column-hive should support DECIMAL(10,5) format.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
Support object store
Hi Expert, Does Sqoop support object store? For example Amazon S3? Angela
[jira] [Updated] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[
https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Lin updated SQOOP-3152:
Attachment: SQOOP-3152.patch
Patch added.
> --map-column-hive to support DECIMAL(xx,xx)
> ---
>
> Key: SQOOP-3152
> URL: https://issues.apache.org/jira/browse/SQOOP-3152
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: SQOOP-3152.patch
>
>
> The following command:
> sqoop import --connect jdbc:mysql://localhost/test --username root --password
> 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver
> --verbose --hive-import --hive-database default --hive-table decimal_table
> --hive-overwrite --map-column-hive a='DECIMAL(10,4)'
> will fail with below error:
> {code}
> 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping
> should be the form key=value[,key=value]*
> java.lang.IllegalArgumentException: Malformed mapping. Column mapping should
> be the form key=value[,key=value]*
> at
> org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333)
> at
> org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349)
> at
> org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198)
> at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011)
> at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:135)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> Malformed mapping. Column mapping should be the form key=value[,key=value]*
> {code}
> --map-column-hive should support DECIMAL(10,5) format.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (SQOOP-3152) --map-column-hive to support DECIMAL(xx,xx)
[
https://issues.apache.org/jira/browse/SQOOP-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15923260#comment-15923260
]
Eric Lin commented on SQOOP-3152:
-
After the patch, you will still see the following WARNING message:
{code}
17/03/14 10:53:53 WARN hive.TableDefWriter: Column a had to be cast to a less
precise type in Hive
{code}
This will be addressed in SQOOP-2272
> --map-column-hive to support DECIMAL(xx,xx)
> ---
>
> Key: SQOOP-3152
> URL: https://issues.apache.org/jira/browse/SQOOP-3152
> Project: Sqoop
> Issue Type: Bug
>Affects Versions: 1.4.6
>Reporter: Eric Lin
>Assignee: Eric Lin
>Priority: Minor
> Attachments: SQOOP-3152.patch
>
>
> The following command:
> sqoop import --connect jdbc:mysql://localhost/test --username root --password
> 'cloudera' --table decimal_table -m 1 --driver com.mysql.jdbc.Driver
> --verbose --hive-import --hive-database default --hive-table decimal_table
> --hive-overwrite --map-column-hive a='DECIMAL(10,4)'
> will fail with below error:
> {code}
> 17/03/13 18:42:09 DEBUG sqoop.Sqoop: Malformed mapping. Column mapping
> should be the form key=value[,key=value]*
> java.lang.IllegalArgumentException: Malformed mapping. Column mapping should
> be the form key=value[,key=value]*
> at
> org.apache.sqoop.SqoopOptions.parseColumnMapping(SqoopOptions.java:1333)
> at
> org.apache.sqoop.SqoopOptions.setMapColumnHive(SqoopOptions.java:1349)
> at
> org.apache.sqoop.tool.BaseSqoopTool.applyHiveOptions(BaseSqoopTool.java:1198)
> at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:1011)
> at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
> at org.apache.sqoop.Sqoop.run(Sqoop.java:135)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
> at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
> at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
> Malformed mapping. Column mapping should be the form key=value[,key=value]*
> {code}
> --map-column-hive should support DECIMAL(10,5) format.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
