Eylon Ronen created SPARK-47660:
-----------------------------------
Summary: Memory leak in StructType column's .dropFields method
Key: SPARK-47660
URL: https://issues.apache.org/jira/browse/SPARK-47660
Project: Spark
Issue Type: Bug
Components: Java API, SQL
Affects Versions: 3.5.1
Reporter: Eylon Ronen
We noticed that when we're using the .dropFields method, for example by calling
df.withColum(column, col(column).dropFields("fieldToDrop")) the heap
"explodes". we noticed it when our integration tests, for which 512MB heap is
always enough started crashing with OutOfMemoryException and after a lot of
trial and error we found that when we remove the .dropFields method the heap
goes back downÂ
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
