[ https://issues.apache.org/jira/browse/SPARK-28613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
angerszhu updated SPARK-28613: ------------------------------ Description: When we run action DataFrame.collect() , for the configuration *spark.driver.maxResultSize ,*when determine if the returned data exceeds the limit, it will use the compressed byte array's size, it it not > Spark SQL action collect just judge size of compressed RDD's size, not > accurate enough > -------------------------------------------------------------------------------------- > > Key: SPARK-28613 > URL: https://issues.apache.org/jira/browse/SPARK-28613 > Project: Spark > Issue Type: Wish > Components: SQL > Affects Versions: 2.4.0 > Reporter: angerszhu > Priority: Major > > When we run action DataFrame.collect() , for the configuration > *spark.driver.maxResultSize ,*when determine if the returned data exceeds > the limit, it will use the compressed byte array's size, it it not -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org