Gaurav Shah created SPARK-17593:

             Summary: list files on s3 very slow
                 Key: SPARK-17593
             Project: Spark
          Issue Type: Bug
    Affects Versions: 2.0.0
         Environment: spark 2.0.0, hadoop 2.7.2 ( hadoop 2.7.3)
            Reporter: Gaurav Shah

lets say we have following partitioned data:
  -- event_date=2015-01-01
    -- event_hour=2015-01-1
      -- part10000.parquet.gz
  -- event_date=2015-01-02
    -- event_hour=5
      -- part10000.parquet.gz
To read (or write ) parquet partitioned data via spark it makes call to 
`ListingFileCatalog.listLeafFiles` .  Which recursively tries to list all files 
and folders.

In this case if we had 300 dates, we would have created 300 jobs each trying to 
get filelist from date_directory. This process takes about 10 minutes to finish 
( with 2 executors). vs if I use a ruby script to get list of all files 
recursively in the same folder it takes about 1 minute, on the same machine 
with just 1 thread. 

I am confused as to why it would take so much time extra for listing files.
spark code:
val sparkSession = org.apache.spark.sql.SparkSession.builder
.config("spark.sql.parquet.filterPushdown", true)
.config("spark.sql.hive.verifyPartitionPath", false)

val df ="mergeSchema","false").format("parquet").load("s3n://bucket_name/events_v3")
        |select verb,count(*) from temp_events where event_date = "2016-08-05" 
group by verb

ruby code:
gem 'aws-sdk', '~> 2'
require 'aws-sdk'
client =>'us-west-1')
next_continuation_token = nil
total = 0
loop do
a= client.list_objects_v2({
  bucket: "bucket", # required
  max_keys: 1000,
  prefix: "events_v3/",
  continuation_token: next_continuation_token ,
  fetch_owner: false,
puts a.contents.last.key
total += a.contents.size
next_continuation_token = a.next_continuation_token
break unless a.is_truncated

puts "total"
puts total

tried looking into following bug:
but hadoop 2.7.3 doesn't solve that problem
stackoverflow reference:

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to