Gengliang Wang created SPARK-23896:

             Summary: Improve PartitioningAwareFileIndex
                 Key: SPARK-23896
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.3.1
            Reporter: Gengliang Wang

Currently `PartitioningAwareFileIndex` accepts an optional parameter 
`userPartitionSchema`. If provided, it will combine the inferred partition 
schema with the parameter.


1. to get the inferred partition schema, we have to create a temporary file 

2. to get `userPartitionSchema`, we need to  combine inferred partition schema 
with `userSpecifiedSchema` 

Only after that, a final version of `PartitioningAwareFileIndex` is created.


This can be improved by passing `userSpecifiedSchema` to 

With the improvement, we can reduce redundant code and avoid parsing the file 
partition twice. 


This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to