[
https://issues.apache.org/jira/browse/SPARK-47660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eylon Ronen updated SPARK-47660:
--------------------------------
Priority: Major (was: Minor)
> Memory leak in StructType column's .dropFields method
> -----------------------------------------------------
>
> Key: SPARK-47660
> URL: https://issues.apache.org/jira/browse/SPARK-47660
> Project: Spark
> Issue Type: Bug
> Components: Java API, SQL
> Affects Versions: 3.5.1
> Reporter: Eylon Ronen
> Priority: Major
>
> We noticed that when we're using the .dropFields method, for example by
> calling
> df.withColum(column, col(column).dropFields("fieldToDrop")) the heap
> "explodes". we noticed it when our integration tests, for which 512MB heap is
> always enough started crashing with OutOfMemoryException (even when we
> increased the heap size to 8GB) and after a lot of trial and error we found
> that when we remove the .dropFields method the heap goes back downÂ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
