Robert Kruszewski created SPARK-19213:

             Summary: FileSourceScanExec usese sparksession from 
hadoopfsrelation creation time instead of the one active at time of execution
                 Key: SPARK-19213
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.1.0
            Reporter: Robert Kruszewski

If you look at
 you'll notice that the sparksession used for execution is the one that was 
captured from logicalplan. Whereas in other places you have
 and SparkPlan captures active session upon execution in

>From my understanding of the io code it would be beneficial to be able to use 
>the active session in order to be able to modify hadoop config without 
>recreating the dataset. What would be interesting is to not lock the spark 
>session in the physical plan for ios and let you share datasets across spark 
>sessions. Is that supposed to work? Otherwise you'd have to get a new query 
>execution to bind to new sparksession which would only let you share logical 

I am sending pr along with the latter.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to