[ https://issues.apache.org/jira/browse/SPARK-17986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Drew Robb updated SPARK-17986: ------------------------------ Description: The SQLTransformer creates a temporary table when called, and does not delete this temporary table. When using a SQLTransformer in a long running Spark Streaming task, these temporary tables accumulate. I believe that the fix would be as simple as calling `dataset.sparkSession.catalog.dropTempView(tableName)` in the last part of `transform`: https://github.com/apache/spark/blob/v2.0.1/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala#L65. was: The SQLTransformer creates a temporary table when called, and does not delete this temporary table. When using a SQLTransformer in a long running Spark Streaming task, these temporary tables accumulate. I believe that the fix would be as simple as calling `dataset.sparkSession.catalog.dropTempView(tableName)` in the last part of `transform`: https://github.com/apache/spark/blob/v2.0.1/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala#L65. I would be happy to attempt this fix myself if someone could validate this issue. > SQLTransformer leaks temporary tables > ------------------------------------- > > Key: SPARK-17986 > URL: https://issues.apache.org/jira/browse/SPARK-17986 > Project: Spark > Issue Type: Bug > Components: ML > Affects Versions: 2.0.1 > Reporter: Drew Robb > Priority: Minor > > The SQLTransformer creates a temporary table when called, and does not delete > this temporary table. When using a SQLTransformer in a long running Spark > Streaming task, these temporary tables accumulate. > I believe that the fix would be as simple as calling > `dataset.sparkSession.catalog.dropTempView(tableName)` in the last part of > `transform`: > https://github.com/apache/spark/blob/v2.0.1/mllib/src/main/scala/org/apache/spark/ml/feature/SQLTransformer.scala#L65. > -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org