[
https://issues.apache.org/jira/browse/SPARK-50768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wuyi updated SPARK-50768:
-------------------------
Description:
SPAKR-49980 tried to fix this kind of issue by applying
{code:java}
def tryInitializeResource[R <: Closeable, T](createResource: => R)(initialize:
R => T): T = {
val resource = createResource
try {
initialize(resource)
} catch {
case e: Throwable =>
resource.close()
throw e
}
} {code}
when creating resources. This utility function has an issue that `resource`
could not be completed initiliazed if interruption happens during
`initialize(resource)`. As a result, `resource.close()` could not completely
clean up intermediate created resources underlying and leads to resource leak.
was:
SPAKR-49980 tried to fix this kind of issue by applying
{code:java}
def tryInitializeResource[R <: Closeable, T](createResource: => R)(initialize:
R => T): T = {
val resource = createResource
try {
initialize(resource)
} catch {
case e: Throwable =>
resource.close()
throw e
}
} {code}
when creating resources. This utility function has an issue that it assumes the
task interruption happpens exactly during the `initialize(resource)`. But it is
not always true. And so the issue actully still exits.
> Pontential file stream leaks caused by task thread interruption
> ---------------------------------------------------------------
>
> Key: SPARK-50768
> URL: https://issues.apache.org/jira/browse/SPARK-50768
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.3
> Reporter: wuyi
> Priority: Major
>
> SPAKR-49980 tried to fix this kind of issue by applying
> {code:java}
> def tryInitializeResource[R <: Closeable, T](createResource: =>
> R)(initialize: R => T): T = {
> val resource = createResource
> try {
> initialize(resource)
> } catch {
> case e: Throwable =>
> resource.close()
> throw e
> }
> } {code}
> when creating resources. This utility function has an issue that `resource`
> could not be completed initiliazed if interruption happens during
> `initialize(resource)`. As a result, `resource.close()` could not completely
> clean up intermediate created resources underlying and leads to resource leak.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
