Github user rtreffer commented on a diff in the pull request:
https://github.com/apache/spark/pull/6796#discussion_r33420719
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/parquet/CatalystSchemaConverter.scala
---
@@ -169,11 +169,12 @@ private[parquet] class CatalystSchemaConverter(
}
case INT96 =>
- CatalystSchemaConverter.analysisRequire(
- assumeInt96IsTimestamp,
- "INT96 is not supported unless it's interpreted as timestamp. " +
- s"Please try to set ${SQLConf.PARQUET_INT96_AS_TIMESTAMP.key}
to true.")
- TimestampType
+ field.getOriginalType match {
+ case DECIMAL => makeDecimalType(maxPrecisionForBytes(12))
+ case _ if assumeInt96IsTimestamp => TimestampType
+ case null => makeDecimalType(maxPrecisionForBytes(12))
+ case _ => illegalType()
+ }
--- End diff --
Didn't know about the deprecation, will drop it.
Am 28. Juni 2015 10:56:00 MESZ, schrieb Cheng Lian
<[email protected]>:
>> @@ -169,11 +169,12 @@ private[parquet] class CatalystSchemaConverter(
>> }
>>
>> case INT96 =>
>> - CatalystSchemaConverter.analysisRequire(
>> - assumeInt96IsTimestamp,
>> - "INT96 is not supported unless it's interpreted as
>timestamp. " +
>> - s"Please try to set
>${SQLConf.PARQUET_INT96_AS_TIMESTAMP.key} to true.")
>> - TimestampType
>> + field.getOriginalType match {
>> + case DECIMAL => makeDecimalType(maxPrecisionForBytes(12))
>> + case _ if assumeInt96IsTimestamp => TimestampType
>> + case null => makeDecimalType(maxPrecisionForBytes(12))
>> + case _ => illegalType()
>> + }
>
>`INT96` is only used for nanosecond timestamp types for historical
>reasons, and is to be deprecated. Let's not use it for decimals.
>
>---
>Reply to this email directly or view it on GitHub:
>https://github.com/apache/spark/pull/6796/files#r33420609
--
Diese Nachricht wurde von meinem Android-Mobiltelefon mit K-9 Mail gesendet.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
