[
https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129327#comment-16129327
]
Robert Kruszewski commented on SPARK-20952:
-------------------------------------------
I don't care about what actual filesystem impls are doing. This is because I
have a wrapper that I want to have a handle on actual taskcontext that extracts
the right parameters and delegates to the underlying system. As such anything
underneath my wrapper doesn't have to deal with spark. We have been relying on
taskcontext being available everywhere on the executor during execution.
> ParquetFileFormat should forward TaskContext to its forkjoinpool
> ----------------------------------------------------------------
>
> Key: SPARK-20952
> URL: https://issues.apache.org/jira/browse/SPARK-20952
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: Robert Kruszewski
> Priority: Minor
>
> TaskContext is a ThreadLocal as a result when you fork a thread inside your
> executor task you lose the handle on the original context set by the
> executor. We should change it to InheritableThreadLocal so we can access it
> inside thread pools on executors.
> See ParquetFileFormat#readFootersInParallel for example of code that uses
> thread pools inside the tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
