In our application we are writing a parquet file as per the specification
for the Decimal handling here
<https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal>
and mapping the Decimal values with precision < 9 to INT32 and Precision <
18 to INT64.
While doing this, we set the length to the actual precision that is defined
on the field or the column.
*Sample Code Snippet*
if (precision <= 9) {
DecimalPrimitiveType = PrimitiveTypeName.INT32;
}
else if (precision <= 18) {
DecimalPrimitiveType = PrimitiveTypeName.INT64;
}
else {
DecimalPrimitiveType = PrimitiveTypeName.FIXED_LEN_BYTE_ARRAY;
precision = 16;
}
field.isNullable() ? Types.optional(DecimalPrimitiveType).
*length(precision)* : Types.required(DecimalPrimitiveType))
.*length(precision)*.as(LogicalTypeAnnotation.decimalType(field.getScale(),
field.getPrecision())).named(fieldName);
Essentially, when the decimal type is being mapped to INT32, then length is
set to the precision defined on the field. When I create a hive table based
on this Parquet file and try to read the data from the hive table, hive is
choking and returning the following error.
*Column precision <= 9*
Error: java.io.IOException: java.lang.IllegalStateException: Group type
[message default {
required int32 decimal_col_6_3 (DECIMAL(6,3));
*Column precision <= 18*
Error: java.io.IOException: java.lang.IllegalStateException: Group type
[message default {
required int64 decimal_col_15_3 (DECIMAL(15,3));
When we set the length to 0 while mapping the Decimal to INT32 or INT64,
then hive is able to successfully read the data.
We would like to understand what should be the length set to when the
Decimal is being mapped to INT32 or INT64 ? The Parquet Specification does
not mention anything about that.
Could someone please provide some guidance or pointers on that ?
--
Srinivas
(*-*)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
You have to grow from the inside out. None can teach you, none can make you
spiritual.
-Narendra Nath Dutta(Swamy Vivekananda)
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
