Burak Yavuz created SPARK-17613: ----------------------------------- Summary: PartitioningAwareFileCatalog.allFiles doesn't handle URI specified path at parent Key: SPARK-17613 URL: https://issues.apache.org/jira/browse/SPARK-17613 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.0.0 Reporter: Burak Yavuz
Consider you have a bucket as {code} s3a://some-bucket {code} and under it you have files: {code} s3a://some-bucket/file1.parquet s3a://some-bucket/file2.parquet {code} Getting the parent path of {code}s3a://some-bucket/file1.parquet{code} yields {code}s3a://some-bucket/{code} and the ListingFileCatalog uses this as the key in the hash map. When catalog.allFiles is called, we use {code}s3a://some-bucket{code} (no slash at the end) to get the list of files, and we're left with an empty list! -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org