[ https://issues.apache.org/jira/browse/SQOOP-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16358981#comment-16358981 ]
Greg Lindholm edited comment on SQOOP-3285 at 2/9/18 9:26 PM: -------------------------------------------------------------- Patch is attached [^0001-Fix-DECIMAL-percision-loss-with-Hcatalog-imports.patch] A couple lines added to org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.convertNumberTypes() The fix is for first check if the value is a BigDecimal or a BigInteger and if so use the value to create the HiveDecimal directly to avoid converting it to a Double. Here is the new logic to convert a Number into a HiveDecimal. {code:java} } else if (hfsType == HCatFieldSchema.Type.DECIMAL) { if (n instanceof BigDecimal) { return HiveDecimal.create((BigDecimal)n); } if (n instanceof BigInteger) { return HiveDecimal.create((BigInteger)n); } BigDecimal bd = new BigDecimal(n.doubleValue(), MathContext.DECIMAL128); return HiveDecimal.create(bd); } {code} was (Author: glindholm): Patch is attached [^0001-Fix-DECIMAL-percision-loss-with-Hcatalog-imports.patch] A couple lines added to org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.convertNumberTypes() The fix is for first check if the value is a BigDecimal or a BigInteger and if so use the value to create the HiveDecimal directly to avoid converting it to a Double. Here is the logic to new convert a Number into a HiveDecimal. {code:java} } else if (hfsType == HCatFieldSchema.Type.DECIMAL) { if (n instanceof BigDecimal) { return HiveDecimal.create((BigDecimal)n); } if (n instanceof BigInteger) { return HiveDecimal.create((BigInteger)n); } BigDecimal bd = new BigDecimal(n.doubleValue(), MathContext.DECIMAL128); return HiveDecimal.create(bd); } {code} > Hcatalog import is loosing precision on DECIMAL fields > ------------------------------------------------------ > > Key: SQOOP-3285 > URL: https://issues.apache.org/jira/browse/SQOOP-3285 > Project: Sqoop > Issue Type: Bug > Components: hive-integration > Affects Versions: 1.4.7 > Reporter: Greg Lindholm > Priority: Major > Attachments: > 0001-Fix-DECIMAL-percision-loss-with-Hcatalog-imports.patch > > > Import with hcatalog for DECIMAL fields is loosing precision. > The bug is in > org.apache.sqoop.mapreduce.hcat.SqoopHCatImportHelper.convertNumberTypes() > Internally the BigDecimal value is converted to a Double then back into a > BigDecimal. > As a test I imported from a MySQL db a DECIMAL(38,8) field: > {code:java} > create table sampledata1 ( n38 DECIMAL(38,8) ); > insert into sampledata1 ( n38 ) values ( > -123456789012345678901234567890.12345678 ) > sqoop import \ > --connect jdbc:mysql://10.210.144.22:3306/greg --username greg -P \ > -m 1 \ > --table sampledata1 \ > --hcatalog-table sampledata1_orc2 \ > --create-hcatalog-table \ > --hcatalog-storage-stanza 'stored as orc' > {code} > The result in the hive table does not match the input. > {noformat} > Original: > -123456789012345678901234567890.12345678 > Result: > -123456789012345677877719597056.00000000{noformat} > I get the same result with Orc, RCfile, and Avro files. > > I believe I found the problem and will provide a patch shortly. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)