Github user rtreffer commented on a diff in the pull request:
https://github.com/apache/spark/pull/6796#discussion_r32891515
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/ParquetTypes.scala ---
@@ -229,11 +231,15 @@ private[parquet] object ParquetTypesConverter extends
Logging {
case LongType => Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT64))
case TimestampType =>
Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT96))
case DecimalType.Fixed(precision, scale) if precision <= 18 =>
- // TODO: for now, our writer only supports decimals that fit in a
Long
Some(ParquetTypeInfo(ParquetPrimitiveTypeName.FIXED_LEN_BYTE_ARRAY,
--- End diff --
What would we gain by encoding it that way?
We use a minimal length fixed byte array which should provide a similar
compact encoding. (DECIMAL(9) should end up as 4 bytes, and smaller decimal
values should take even less space)
Decoding is a different story, though.
PS: I was focusing on DECIMAL with precision >=19. Shouldn't small decimal
handling be a new ticket?
Am 21. Juni 2015 02:41:44 MESZ, schrieb Davies Liu
<[email protected]>:
>> @@ -229,11 +231,15 @@ private[parquet] object ParquetTypesConverter
>extends Logging {
>> case LongType =>
>Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT64))
>> case TimestampType =>
>Some(ParquetTypeInfo(ParquetPrimitiveTypeName.INT96))
>> case DecimalType.Fixed(precision, scale) if precision <= 18 =>
>> - // TODO: for now, our writer only supports decimals that fit
>in a Long
>>
>Some(ParquetTypeInfo(ParquetPrimitiveTypeName.FIXED_LEN_BYTE_ARRAY,
>
>We should int32, int64 if possible, see
>https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#decimal
>```
>DECIMAL can be used to annotate the following types:
>
>int32: for 1 <= precision <= 9
>int64: for 1 <= precision <= 18; precision <= 10 will produce a warning
>fixed_len_byte_array: precision is limited by the array size. Length n
>can store <= floor(log_10(2^(8*n - 1) - 1)) base-10 digits
>binary: precision is not limited, but is required. The minimum number
>of bytes to store the unscaled value should be used.
>```
>
>---
>Reply to this email directly or view it on GitHub:
>https://github.com/apache/spark/pull/6796/files#r32889441
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
