Li Yuanjian created SPARK-21435:

             Summary: Empty files should be skipped while write to file
                 Key: SPARK-21435
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.2.0
            Reporter: Li Yuanjian

Consider of this scenario, source table has many partitions and data files, 
after the query filter, only a few data write to the destination dir.
In this case the destination dir or table will have many empty files or files 
only have schema meta(parquet format), I know we can use coalesce but skip the 
empty file may be more better in this case.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to