[
https://issues.apache.org/jira/browse/SPARK-20406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-20406.
-------------------------------
Resolution: Duplicate
Search JIRA first please
> Upgrade parquet to 1.9
> ----------------------
>
> Key: SPARK-20406
> URL: https://issues.apache.org/jira/browse/SPARK-20406
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Kaspar Sakmann
> Labels: maven
>
> Please upgrade parquet to 1.9.
> with 500G memory 48 CPUs and a dataset about 1G in size
> using Parquet 1.8 causes warnings:
> "Total allocation exceeds 95.00% (2,147,483,647 bytes) of heap memory
> Scaling row group sizes to 34.04% for 47 writers"
> The issue appears to have been fixed with parquet 1.9, see
> https://issues.apache.org/jira/browse/PARQUET-340
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
