This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new caeaeb22ccb3 [SPARK-58382][SQL] binaryFile archive support via 
wholeFile option
caeaeb22ccb3 is described below

commit caeaeb22ccb3bbba6d02263c62156296e16f540f
Author: akshatshenoi-db <[email protected]>
AuthorDate: Tue Jul 28 22:20:11 2026 +0800

    [SPARK-58382][SQL] binaryFile archive support via wholeFile option
    
    ### What changes were proposed in this pull request?
    
    This PR lets `format("binaryFile")` read tar/zip/7z archives, extending the 
archive-reader
    series (SPARK-57135 CSV, SPARK-57419 JSON, SPARK-57478 text, SPARK-57479 
XML, SPARK-57481 Avro).
    It is gated by a new binaryFile option `wholeFile` (default `true`) and the 
existing
    `spark.sql.files.archive.reader.enabled` flag (default `false`).
    
    - `wholeFile=true` (default): the archive is read as a single record (its 
raw bytes) -- unchanged
      from today's non-archive behavior.
    - `wholeFile=false`: one row is emitted per inner entry. `content` holds 
the entry's unpacked
      bytes, `path` is `<archive>!/<entryName>`, `length` is the entry's size, 
and `modificationTime`
      is the parent archive's (archive entry timestamps are optional, so the 
parent's is used).
    
    `BinaryFileFormat` mixes in `SupportsArchiveFormat`. On the 
`wholeFile=false` archive path it
    streams each entry via `SupportsArchiveFormat.readArchiveEntries` and 
builds one row per entry from
    a synthetic per-entry `FileStatus`, reusing the same row-builder 
(`BinaryFileFormat.parse`) as the
    non-archive path so length/modtime filter pushdown applies per entry.
    
    To expose the entry's size to consumers, the `readArchiveEntries` 
`parseEntry` callback now
    receives the `ArchiveEntry` instead of just the entry name; the existing 
CSV and `localizeEntries`
    callers are updated to read `entry.getName`.
    
    Known limitation (zip, until `ZipFile`): the streaming 
`ZipArchiveInputStream` reports an entry's
    `getSize` as `-1` when the size is only in a trailing data descriptor, so 
for such entries the
    per-entry `length` column is `-1` and the 
`spark.sql.sources.binaryFile.maxLength` guard
    (`status.getLen > maxLength`) is skipped, i.e. the entry is read unbounded. 
tar and 7z report real
    sizes and are unaffected. This closes when zip reads move to `ZipFile`, 
which exposes entry sizes
    from the central directory up front.
    
    ### Why are the changes needed?
    
    binaryFile is the natural way to ingest opaque/blob files, and archives 
(tar/zip/7z) are a common
    packaging for large numbers of such files. This lets an archive be read as 
a directory of its
    entries without unpacking it to disk first, completing the archive-reader 
feature across the
    file-based formats.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, gated behind `spark.sql.files.archive.reader.enabled` (default 
`false`). When enabled,
    `spark.read.format("binaryFile").option("wholeFile", 
"false").load(<archive>)` returns one row per
    inner entry. With the flag off, or with `wholeFile=true` (the default), 
behavior is unchanged.
    
    ### How was this patch tested?
    
    New `BinaryFileArchiveReadBase` with `BinaryFileTarArchiveReadSuite`,
    `BinaryFileZipArchiveReadSuite`, and `BinaryFileSevenZArchiveReadSuite`, 
covering: whole-file vs
    per-entry reads, per-entry content/path/length, empty archives, 
hidden-entry skipping,
    `spark.sql.sources.binaryFile.maxLength` enforcement, length filter 
pushdown, single-partition
    scans, and corrupt-archive handling under `ignoreCorruptFiles`. The 
per-entry `length` assertions
    are skipped for zip because the streaming `ZipArchiveInputStream` reports a 
data-descriptor entry's
    size as `-1`; they re-enable when zip reads move to `ZipFile`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code
    
    Closes #57578 from akshatshenoi-db/archive-binaryfile.
    
    Authored-by: akshatshenoi-db <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
---
 .../datasources/SupportsArchiveFormat.scala        |  11 +-
 .../datasources/binaryfile/BinaryFileFormat.scala  |  89 +++++---
 .../execution/datasources/csv/CSVDataSource.scala  |   4 +-
 .../datasources/BinaryFileArchiveReadBase.scala    | 234 +++++++++++++++++++++
 .../datasources/SupportsArchiveFormatSuite.scala   |  32 +--
 5 files changed, 323 insertions(+), 47 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
index a54e83d9e83f..1c6c63582039 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
@@ -272,21 +272,21 @@ object SupportsArchiveFormat {
 
   /**
    * Streams the archive at `path` entry by entry, applying `parseEntry` to 
each non-skipped
-   * `(name, stream)` and concatenating the results.
+   * `(entry, stream)` and concatenating the results.
    *
    * @param path                    the archive path
    * @param conf                    Hadoop configuration used to open the 
archive
    * @param ignoredPathSegmentRegex per-segment filter for entries to skip 
(defaults to the
    *                                `InMemoryFileIndex` filter); pass a custom 
one to match a
    *                                loose-file scan
-   * @param parseEntry              turns one entry's `(name, stream)` into an 
iterator of results
+   * @param parseEntry              turns one entry's `(entry, stream)` into 
an iterator of results
    * @return the concatenated results across kept entries, lazily one entry at 
a time
    */
   def readArchiveEntries[T](
       path: Path,
       conf: Configuration,
       ignoredPathSegmentRegex: Pattern = 
HadoopFSUtils.defaultIgnoredPathSegmentRegexPattern)(
-      parseEntry: (String, InputStream) => Iterator[T]): Iterator[T] = {
+      parseEntry: (ArchiveEntry, InputStream) => Iterator[T]): Iterator[T] = {
     val archive = openArchiveStream(path, conf)
     var closed = false
 
@@ -322,7 +322,7 @@ object SupportsArchiveFormat {
           } else {
             // CloseShieldInputStream ignores close(), so a parser closing its 
input does not close
             // the archive; any unread remainder is skipped by getNextEntry() 
when advancing.
-            currentIter = parseEntry(entry.getName, 
CloseShieldInputStream.wrap(archive))
+            currentIter = parseEntry(entry, 
CloseShieldInputStream.wrap(archive))
           }
         }
       }
@@ -390,7 +390,8 @@ object SupportsArchiveFormat {
       conf: Configuration,
       localDir: File,
       entryFilter: String => Boolean): Iterator[(String, File)] =
-    readArchiveEntries(path, conf) { (name, in) =>
+    readArchiveEntries(path, conf) { (entry, in) =>
+      val name = entry.getName
       if (entryFilter(name)) {
         Iterator.single((name, copyEntryToLocalFile(in, localDir, name)))
       } else {
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala
index 19b67d0c5390..33de63072ef3 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/binaryfile/BinaryFileFormat.scala
@@ -17,6 +17,7 @@
 
 package org.apache.spark.sql.execution.datasources.binaryfile
 
+import java.io.InputStream
 import java.sql.Timestamp
 
 import com.google.common.io.Closeables
@@ -25,13 +26,15 @@ import org.apache.hadoop.fs.{FileStatus, Path}
 import org.apache.hadoop.mapreduce.Job
 
 import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.FileSourceOptions
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
-import org.apache.spark.sql.catalyst.util.DateTimeUtils
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateTimeUtils}
 import org.apache.spark.sql.errors.QueryExecutionErrors
-import org.apache.spark.sql.execution.datasources.{FileFormat, 
OutputWriterFactory, PartitionedFile}
+import org.apache.spark.sql.execution.datasources.{FileFormat, 
OutputWriterFactory, PartitionedFile, SupportsArchiveFormat}
 import org.apache.spark.sql.internal.SessionStateHelper
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.internal.SQLConf.SOURCES_BINARY_FILE_MAX_LENGTH
 import org.apache.spark.sql.sources.{And, DataSourceRegister, EqualTo, Filter, 
GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual, Not, Or}
 import org.apache.spark.sql.types._
@@ -57,6 +60,7 @@ import org.apache.spark.util.SerializableConfiguration
  * }}}
  */
 case class BinaryFileFormat() extends FileFormat
+  with SupportsArchiveFormat
   with DataSourceRegister with SessionStateHelper {
 
   import BinaryFileFormat._
@@ -104,34 +108,29 @@ case class BinaryFileFormat() extends FileFormat
     val filterFuncs = filters.flatMap(filter => createFilterFunction(filter))
     val maxLength = 
getSqlConf(sparkSession).getConf(SOURCES_BINARY_FILE_MAX_LENGTH)
 
+    // `wholeFile=false` reads an archive as one row per inner entry; the 
default keeps the whole
+    // archive as a single record.
+    val caseInsensitiveOptions = CaseInsensitiveMap(options)
+    val archiveReadEnabled = 
!caseInsensitiveOptions.get(WHOLE_FILE).forall(_.toBoolean) &&
+      getSqlConf(sparkSession).getConf(SQLConf.ARCHIVE_FORMAT_READER_ENABLED)
+
     file: PartitionedFile => {
       val path = file.toPath
       val fs = path.getFileSystem(broadcastedHadoopConf.value.value)
       val status = fs.getFileStatus(path)
-      if (filterFuncs.forall(_.apply(status))) {
-        val writer = new UnsafeRowWriter(requiredSchema.length)
-        writer.resetRowWriter()
-        requiredSchema.fieldNames.zipWithIndex.foreach {
-          case (PATH, i) => writer.write(i, 
UTF8String.fromString(status.getPath.toString))
-          case (LENGTH, i) => writer.write(i, status.getLen)
-          case (MODIFICATION_TIME, i) =>
-            writer.write(i, 
DateTimeUtils.millisToMicros(status.getModificationTime))
-          case (CONTENT, i) =>
-            if (status.getLen > maxLength) {
-              throw 
QueryExecutionErrors.fileLengthExceedsMaxLengthError(status, maxLength)
-            }
-            val stream = fs.open(status.getPath)
-            try {
-              writer.write(i, stream.readAllBytes())
-            } finally {
-              Closeables.close(stream, true)
-            }
-          case (other, _) =>
-            throw QueryExecutionErrors.unsupportedFieldNameError(other)
+      if (archiveReadEnabled && SupportsArchiveFormat.isArchivePath(path)) {
+        val ignoredPathSegmentRegex =
+          new 
FileSourceOptions(caseInsensitiveOptions).ignoredPathSegmentRegexPattern
+        SupportsArchiveFormat.readArchiveEntries(
+            path, broadcastedHadoopConf.value.value, ignoredPathSegmentRegex) 
{ (entry, in) =>
+          val entryStatus = new FileStatus(
+            entry.getSize, false, 0, 0, status.getModificationTime,
+            new Path(s"${status.getPath}!/${entry.getName}"))
+          BinaryFileFormat.parse(() => in, entryStatus, requiredSchema, 
maxLength, filterFuncs)
         }
-        Iterator.single(writer.getRow)
       } else {
-        Iterator.empty
+        BinaryFileFormat.parse(
+          () => fs.open(status.getPath), status, requiredSchema, maxLength, 
filterFuncs)
       }
     }
   }
@@ -144,6 +143,48 @@ object BinaryFileFormat {
   private[binaryfile] val LENGTH = "length"
   private[binaryfile] val CONTENT = "content"
   private[binaryfile] val BINARY_FILE = "binaryFile"
+  private[binaryfile] val WHOLE_FILE = "wholeFile"
+
+  /**
+   * Builds one row from `status` (path/length/modificationTime) and the bytes 
of `getContent`,
+   * skipping the row when `filterFuncs` reject `status`. Used for both a 
standalone file and a
+   * single archive entry (whose `status` describes the entry).
+   */
+  def parse(
+      getContent: () => InputStream,
+      status: FileStatus,
+      schema: StructType,
+      maxLength: Int,
+      filterFuncs: Seq[FileStatus => Boolean]): Iterator[InternalRow] = {
+    if (filterFuncs.forall(_.apply(status))) {
+      val writer = new UnsafeRowWriter(schema.length)
+      writer.resetRowWriter()
+      schema.fieldNames.zipWithIndex.foreach {
+        case (PATH, i) => writer.write(i, 
UTF8String.fromString(status.getPath.toString))
+        case (LENGTH, i) => writer.write(i, status.getLen)
+        case (MODIFICATION_TIME, i) =>
+          writer.write(i, 
DateTimeUtils.millisToMicros(status.getModificationTime))
+        case (CONTENT, i) =>
+          // Skipped when the size is unknown (`status.getLen < 0`), which is 
the case for a
+          // streaming-zip entry sized only by a trailing data descriptor; 
such an entry is read
+          // unbounded until zip reads move to ZipFile (which exposes entry 
sizes up front).
+          if (status.getLen > maxLength) {
+            throw QueryExecutionErrors.fileLengthExceedsMaxLengthError(status, 
maxLength)
+          }
+          val content = getContent()
+          try {
+            writer.write(i, content.readAllBytes())
+          } finally {
+            Closeables.close(content, true)
+          }
+        case (other, _) =>
+          throw QueryExecutionErrors.unsupportedFieldNameError(other)
+      }
+      Iterator.single(writer.getRow)
+    } else {
+      Iterator.empty
+    }
+  }
 
   /**
    * Schema for the binary file data source.
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
index 8d2e904ff27f..2eb0ad44f472 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
@@ -146,9 +146,9 @@ abstract class CSVDataSource extends Serializable with 
Logging with SupportsArch
       parseEntry: (UnivocityParser, CSVHeaderChecker, InputStream) => 
Iterator[InternalRow])
     : Iterator[InternalRow] = {
     SupportsArchiveFormat.readArchiveEntries(
-        file.toPath, conf, ignoredPathSegmentRegex) { (entryName, in) =>
+        file.toPath, conf, ignoredPathSegmentRegex) { (entry, in) =>
       val headerChecker =
-        getHeaderChecker(true, s"CSV archive entry: 
${file.urlEncodedPath}!/$entryName")
+        getHeaderChecker(true, s"CSV archive entry: 
${file.urlEncodedPath}!/${entry.getName}")
       val parser = getParser()
       headerChecker.setHeaderForSingleVariantColumn =
         CSVDataSource.setHeaderForSingleVariantColumn(conf, file, parser)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BinaryFileArchiveReadBase.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BinaryFileArchiveReadBase.scala
new file mode 100644
index 000000000000..bcbfb89d1241
--- /dev/null
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/BinaryFileArchiveReadBase.scala
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+
+import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.sql.{DataFrame, QueryTest, Row}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+
+/**
+ * Reads of binary files packed in archives via the [[SupportsArchiveFormat]] 
path. The `wholeFile`
+ * option (default true) selects the behavior: true keeps today's 
whole-archive-as-one-record
+ * contract; false emits one row per inner entry -- `content` is the entry's 
unpacked bytes, `path`
+ * is `<archive>!/<entryName>`, and `length` is the entry's size 
(`modificationTime` stays the
+ * parent archive's, since entry timestamps are optional).
+ *
+ * Unlike CSV/JSON this does not extend [[ArchiveReadSuiteBase]]: binaryFile's 
four-column row shape
+ * (path/modificationTime/length/content) differs from the text formats' 
shape, so the shared tests
+ * there do not apply.
+ */
+trait BinaryFileArchiveReadBase extends QueryTest with SharedSparkSession {
+
+  /** Archive extensions to exercise; the head is the default. Supplied by the 
container trait. */
+  protected def archiveExtensions: Seq[String]
+
+  /** Writes `entries` (name -> bytes) into the archive at `dest`. From the 
container trait. */
+  protected def writeArchive(dest: File, entries: Seq[(String, Array[Byte])]): 
Unit
+
+  /** Writes bytes that are not a readable archive at `dest`. From the 
container trait. */
+  protected def writeCorruptArchive(dest: File): Unit
+
+  /** Extension of the archive [[writeCorruptArchive]] produces (corruption is 
format-specific). */
+  protected def corruptArchiveExtension: String
+
+  /**
+   * Whether the container reports each entry's size up front. Streaming zip
+   * (`ZipArchiveInputStream`) does not -- an entry sized only by a trailing 
data descriptor reads
+   * back as `-1` -- so the per-entry `length` tests are skipped for zip until 
it moves to
+   * `ZipFile`. tar and 7z report real sizes.
+   */
+  protected def entrySizeKnown: Boolean = true
+
+  override def sparkConf: SparkConf =
+    super.sparkConf.set(SQLConf.ARCHIVE_FORMAT_READER_ENABLED.key, "true")
+
+  private def bytes(s: String): Array[Byte] = 
s.getBytes(StandardCharsets.UTF_8)
+
+  private def withArchiveFile(
+      extension: String = archiveExtensions.head)(f: File => Unit): Unit =
+    withTempDir(dir => f(new File(dir, s"archive.$extension")))
+
+  private def read(path: String, options: Map[String, String]): DataFrame =
+    spark.read.format("binaryFile").options(options).load(path)
+
+  test("wholeFile=true (default) reads the whole archive as a single record") {
+    archiveExtensions.foreach { ext =>
+      withArchiveFile(ext) { archive =>
+        writeArchive(archive, Seq("a.bin" -> bytes("aaa"), "b.bin" -> 
bytes("bb")))
+        val df = read(archive.getCanonicalPath, Map.empty)
+        assert(df.count() == 1L)
+        val row = df.select("path", "length", "content").head()
+        assert(row.getString(0) == archive.toPath.toUri.toString ||
+          row.getString(0).endsWith(archive.getName))
+        assert(row.getLong(1) == archive.length())
+        // The single record is the raw archive bytes, not any one entry's 
bytes.
+        assert(row.getAs[Array[Byte]](2).length == archive.length())
+      }
+    }
+  }
+
+  test("wholeFile=false emits one row per entry with the entry's unpacked 
content") {
+    archiveExtensions.foreach { ext =>
+      withArchiveFile(ext) { archive =>
+        writeArchive(archive, Seq("a.bin" -> bytes("aaa"), "b.bin" -> 
bytes("bbbb")))
+        checkAnswer(
+          read(archive.getCanonicalPath, Map("wholeFile" -> 
"false")).select("content"),
+          Seq(Row(bytes("aaa")), Row(bytes("bbbb"))))
+      }
+    }
+  }
+
+  test("wholeFile=false sources path and length from each entry, modtime from 
the parent") {
+    assume(entrySizeKnown)
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq("a.bin" -> bytes("aaa"), "b.bin" -> 
bytes("bbbb")))
+      val rows = read(archive.getCanonicalPath, Map("wholeFile" -> "false"))
+        .select("path", "length", "modificationTime").collect()
+      assert(rows.length == 2)
+      // path is `<archive>!/<entryName>`, distinct per entry.
+      val paths = rows.map(_.getString(0))
+      assert(paths.forall(_.contains(s"${archive.getName}!/")))
+      assert(paths.map(_.split("!/").last).sorted === Array("a.bin", "b.bin"))
+      // length is the entry's own unpacked size (3 for "aaa", 4 for "bbbb"), 
not the archive's.
+      val byEntry = rows.map(r => r.getString(0).split("!/").last -> 
r.getLong(1)).toMap
+      assert(byEntry === Map("a.bin" -> 3L, "b.bin" -> 4L))
+      // modificationTime stays the parent archive's, identical across entries.
+      assert(rows.map(_.getTimestamp(2)).distinct.length == 1)
+    }
+  }
+
+  test("wholeFile=false on an empty archive yields no rows") {
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq.empty)
+      checkAnswer(read(archive.getCanonicalPath, Map("wholeFile" -> "false")), 
Seq.empty[Row])
+    }
+  }
+
+  test("wholeFile=false skips hidden entries") {
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq(
+        "a.bin" -> bytes("keep"),
+        "_hidden.bin" -> bytes("drop"),
+        ".dotfile.bin" -> bytes("drop")))
+      checkAnswer(
+        read(archive.getCanonicalPath, Map("wholeFile" -> 
"false")).select("content"),
+        Seq(Row(bytes("keep"))))
+    }
+  }
+
+  test("wholeFile=false enforces SOURCES_BINARY_FILE_MAX_LENGTH per entry") {
+    assume(entrySizeKnown)
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq("big.bin" -> bytes("0123456789")))
+      withSQLConf(SQLConf.SOURCES_BINARY_FILE_MAX_LENGTH.key -> "4") {
+        val e = intercept[SparkException] {
+          read(archive.getCanonicalPath, Map("wholeFile" -> "false")).collect()
+        }
+        assert(e.getMessage.contains("exceeds") || e.getCause != null)
+      }
+    }
+  }
+
+  test("wholeFile=false honors length filter pushdown against each entry") {
+    assume(entrySizeKnown)
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq("a.bin" -> bytes("aaa"), "b.bin" -> 
bytes("bbbb")))
+      // Entry lengths are 3 and 4; the filter selects per entry, not against 
the archive size.
+      val df = read(archive.getCanonicalPath, Map("wholeFile" -> "false"))
+      checkAnswer(df.where("length = 3").select("content"), 
Seq(Row(bytes("aaa"))))
+      checkAnswer(
+        df.where("length <= 4").select("content"), Seq(Row(bytes("aaa")), 
Row(bytes("bbbb"))))
+      checkAnswer(df.where("length > 4").select("content"), Seq.empty[Row])
+    }
+  }
+
+  test("wholeFile=false reads both archive entries and loose files in the same 
input") {
+    withTempDir { dir =>
+      val ext = archiveExtensions.head
+      writeArchive(
+        new File(dir, s"data.$ext"),
+        Seq("a.bin" -> bytes("in-archive-a"), "b.bin" -> 
bytes("in-archive-b")))
+      java.nio.file.Files.write(new File(dir, "loose.bin").toPath, 
bytes("loose"))
+      checkAnswer(
+        read(dir.getCanonicalPath, Map("wholeFile" -> 
"false")).select("content"),
+        Seq(Row(bytes("in-archive-a")), Row(bytes("in-archive-b")), 
Row(bytes("loose"))))
+    }
+  }
+
+  test("wholeFile=false with count and no content column reads the right 
number of rows") {
+    withArchiveFile() { archive =>
+      writeArchive(archive, Seq(
+        "a.bin" -> bytes("a"), "b.bin" -> bytes("bb"), "c.bin" -> 
bytes("ccc")))
+      assert(read(archive.getCanonicalPath, Map("wholeFile" -> 
"false")).count() == 3L)
+    }
+  }
+
+  test("an archive always yields a single partition regardless of size") {
+    withArchiveFile() { archive =>
+      val big = bytes("x" * 4096)
+      writeArchive(archive, (0 until 4).map(i => s"part-$i.bin" -> big))
+      withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "1024") {
+        val df = read(archive.getCanonicalPath, Map("wholeFile" -> "false"))
+        assert(df.rdd.getNumPartitions == 1,
+          s"archive should be a single partition; got 
${df.rdd.getNumPartitions}")
+        assert(df.count() == 4L)
+      }
+    }
+  }
+
+  Seq(true, false).foreach { ignoreCorrupt =>
+    test(s"wholeFile=false ignoreCorruptFiles=$ignoreCorrupt controls skipping 
a corrupt archive") {
+      withArchiveFile(corruptArchiveExtension) { archive =>
+        writeCorruptArchive(archive)
+        withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key -> 
ignoreCorrupt.toString) {
+          val df = read(archive.getCanonicalPath, Map("wholeFile" -> "false"))
+          if (ignoreCorrupt) {
+            checkAnswer(df, Seq.empty[Row])
+          } else {
+            intercept[SparkException](df.collect())
+          }
+        }
+      }
+    }
+  }
+}
+
+class BinaryFileTarArchiveReadSuite extends BinaryFileArchiveReadBase with 
TarArchiveTestUtils {
+
+  override protected def corruptArchiveExtension: String = "tar.gz"
+}
+
+class BinaryFileZipArchiveReadSuite extends BinaryFileArchiveReadBase with 
ZipArchiveTestUtils {
+
+  override protected def corruptArchiveExtension: String = "zip"
+
+  // Streaming `ZipArchiveInputStream` cannot report an entry's size before 
reading it, so the
+  // per-entry `length` tests are skipped for zip. Remove this override once 
zip reads move to
+  // `ZipFile`, which exposes entry sizes from the central directory.
+  override protected def entrySizeKnown: Boolean = false
+}
+
+class BinaryFileSevenZArchiveReadSuite
+  extends BinaryFileArchiveReadBase with SevenZArchiveTestUtils {
+
+  override protected def corruptArchiveExtension: String = "7z"
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
index a676eea5ea45..b63b9ac31031 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
@@ -158,8 +158,8 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
   /** Drains every entry into `(name, decodedText)` pairs through 
`SupportsArchiveFormat`. */
   private def collect(file: File): Seq[(String, String)] =
     SupportsArchiveFormat.readArchiveEntries(new Path(file.toURI), new 
Configuration()) {
-      (name, in) =>
-        Iterator.single((name, new String(readAll(in), 
StandardCharsets.UTF_8)))
+      (entry, in) =>
+        Iterator.single((entry.getName, new String(readAll(in), 
StandardCharsets.UTF_8)))
     }.toList
 
   // ----- isArchivePath ------------------------------------------------------
@@ -256,8 +256,8 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
       // HadoopFSUtils.shouldFilterOutPathName still apply -- mirroring a 
loose-file listing with
       // the ignoredPathSegmentRegex option set to the same regex.
       val entries = SupportsArchiveFormat.readArchiveEntries(
-        new Path(tar.toURI), new Configuration(), Pattern.compile("(?!)")) { 
(name, in) =>
-        Iterator.single((name, new String(readAll(in), 
StandardCharsets.UTF_8)))
+        new Path(tar.toURI), new Configuration(), Pattern.compile("(?!)")) { 
(entry, in) =>
+        Iterator.single((entry.getName, new String(readAll(in), 
StandardCharsets.UTF_8)))
       }.toList
       assert(entries == Seq("_SUCCESS" -> "marker", "real.csv" -> "kept"))
     }
@@ -272,9 +272,9 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
       // parseEntry yields a single element without reading the stream, so 
each invocation maps to
       // exactly one consumed output element -- letting us observe when the 
next entry is opened.
       val it = SupportsArchiveFormat.readArchiveEntries(new Path(tar.toURI), 
new Configuration()) {
-        (name, _) =>
-          opened += name
-          Iterator.single(name)
+        (entry, _) =>
+          opened += entry.getName
+          Iterator.single(entry.getName)
       }
 
       // Construction opens only the first entry; later entries open on demand 
as iteration
@@ -300,11 +300,11 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
 
       val seen = ArrayBuffer[String]()
       val it = SupportsArchiveFormat.readArchiveEntries(new Path(tar.toURI), 
new Configuration()) {
-        (name, in) =>
+        (entry, in) =>
           val body = new String(readAll(in), StandardCharsets.UTF_8)
           in.close() // must NOT close the underlying archive
           seen += body
-          Iterator.single(name)
+          Iterator.single(entry.getName)
       }
       assert(it.toList == List("a.csv", "b.csv"))
       assert(seen.toList == List("a", "b"))
@@ -317,7 +317,7 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
       writeTar(tar, Seq(textEntry("a.csv", "a"), textEntry("b.csv", "b")))
 
       val it = SupportsArchiveFormat.readArchiveEntries(new Path(tar.toURI), 
new Configuration()) {
-        (name, _) => Iterator.single(name)
+        (entry, _) => Iterator.single(entry.getName)
       }
       assert(it.hasNext)
       it.asInstanceOf[Closeable].close()
@@ -346,7 +346,7 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
       try {
         val it = SupportsArchiveFormat.readArchiveEntries(
           new Path(tar.toURI), new Configuration()) {
-          (name, _) => Iterator.single(name)
+          (entry, _) => Iterator.single(entry.getName)
         }
         assert(it.hasNext)
         it.next() // open the archive and register the completion listener
@@ -417,9 +417,9 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
 
       val opened = ArrayBuffer[String]()
       val it = SupportsArchiveFormat.readArchiveEntries(new Path(zip.toURI), 
new Configuration()) {
-        (name, _) =>
-          opened += name
-          Iterator.single(name)
+        (entry, _) =>
+          opened += entry.getName
+          Iterator.single(entry.getName)
       }
       // Construction opens only the first entry; advancing past each boundary 
opens the next.
       assert(opened.toList == List("a.csv"))
@@ -442,11 +442,11 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
 
       val seen = ArrayBuffer[String]()
       val it = SupportsArchiveFormat.readArchiveEntries(new Path(zip.toURI), 
new Configuration()) {
-        (name, in) =>
+        (entry, in) =>
           val body = new String(readAll(in), StandardCharsets.UTF_8)
           in.close() // must NOT close the underlying archive
           seen += body
-          Iterator.single(name)
+          Iterator.single(entry.getName)
       }
       assert(it.toList == List("a.csv", "b.csv"))
       assert(seen.toList == List("a", "b"))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to