[
https://issues.apache.org/jira/browse/SPARK-4768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14295663#comment-14295663
]
Yin Huai commented on SPARK-4768:
---------------------------------
[~taiji] Seems the attached parquet file only has the last row ('test row 4',
null). Can you upload the correct file? Also, can you add a row with a
nanosecond precision timestamp value?
> Add Support For Impala Encoded Timestamp (INT96)
> ------------------------------------------------
>
> Key: SPARK-4768
> URL: https://issues.apache.org/jira/browse/SPARK-4768
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Pat McDonough
> Priority: Critical
> Attachments: 5e4481a02f951e29-651ee94ed14560bf_922627129_data.0.parq
>
>
> Impala is using INT96 for timestamps. Spark SQL should be able to read this
> data despite the fact that it is not part of the spec.
> Perhaps adding a flag to act like impala when reading parquet (like we do for
> strings already) would be useful.
> Here's an example of the error you might see:
> {code}
> Caused by: java.lang.RuntimeException: Potential loss of precision: cannot
> convert INT96
> at scala.sys.package$.error(package.scala:27)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.toPrimitiveDataType(ParquetTypes.scala:61)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.toDataType(ParquetTypes.scala:113)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:314)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$$anonfun$convertToAttributes$1.apply(ParquetTypes.scala:311)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.convertToAttributes(ParquetTypes.scala:310)
> at
> org.apache.spark.sql.parquet.ParquetTypesConverter$.readSchemaFromFile(ParquetTypes.scala:441)
> at
> org.apache.spark.sql.parquet.ParquetRelation.<init>(ParquetRelation.scala:66)
> at org.apache.spark.sql.SQLContext.parquetFile(SQLContext.scala:141)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
