This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3028c1e481c4 [SPARK-58246][SQL] Read files from 7z archives
3028c1e481c4 is described below
commit 3028c1e481c447f03ef1119d0233df2fef5bd020
Author: akshatshenoi-db <[email protected]>
AuthorDate: Wed Jul 22 11:08:54 2026 +0800
[SPARK-58246][SQL] Read files from 7z archives
### What changes were proposed in this pull request?
This adds **7z** (`.7z`) as a third archive container to the archive-reader
feature (gated by
`spark.sql.files.archive.reader.enabled`, default off), alongside tar and
zip. It continues the
series: SPARK-57135/57321 CSV, SPARK-57419 JSON, SPARK-57478 text,
SPARK-57479 XML, SPARK-57481
Avro, SPARK-57705 zip container, SPARK-58110 the `SupportsArchiveFormat`
trait, SPARK-57590 Parquet.
The container engine is format-agnostic: data sources dispatch through the
trait's `isArchivePath`
+ `readArchiveEntries`, so CSV, JSON, text, XML, and Avro get 7z with no
per-data-source change.
Unlike tar and zip, 7z keeps its entry index at the *end* of the file and
can compress entries
together, so it cannot be read from a forward-only stream. The port:
- adds a read-only `HadoopSeekableByteChannel` over `FSDataInputStream` so
`SevenZFile` can seek to
the footer index in place (the archive is never copied to local disk);
- wraps `SevenZFile` in a `SevenZArchiveInputStream` adapter that fits the
existing
`ArchiveInputStream` cursor the engine already consumes, so
`openArchiveStream` gains a single
`.7z` case and the streaming engine itself is unchanged;
- adds `.7z` to `isArchivePath` and the unsupported-archive error message.
commons-compress reads 7z but treats its xz (LZMA/LZMA2) codec as an
optional dependency, so
`org.tukaani:xz` (already version-managed in the parent POM) is declared in
`sql/core/pom.xml`.
It also consolidates the per-container leaf test suites into each format's
`*ArchiveReadBase` (and
merges the CSV header/headerless bases), so adding a container no longer
multiplies suite files;
each format gains a `<Format>SevenZArchiveReadSuite` beside its existing
tar/zip suites.
### Why are the changes needed?
7z is a common archive format for shipping many small files compressed
together. Extending the
archive reader to 7z lets users read tarred/zipped/7z'd data without
unpacking first, and exercises
the seekable-container path (the previous containers were all forward-only
streams).
### Does this PR introduce _any_ user-facing change?
Yes, gated by `spark.sql.files.archive.reader.enabled` (default `false`).
When enabled, reading a
path ending in `.7z` returns the rows of its entries, the same as an
equivalent directory; when
disabled (the default), behavior is unchanged.
### How was this patch tested?
Added a `<Format>SevenZArchiveReadSuite` for CSV (header and headerless),
JSON, text, XML, and Avro
(each binding the format to the new
`SevenZArchiveReadBase`/`SevenZArchiveTestUtils`), so 7z runs the
full format-agnostic and format-specific archive test matrix. Added 7z
cases to
`SupportsArchiveFormatSuite` covering `.7z` dispatch, entry ordering, and
directory/dotfile/marker
skipping through the seek-based reader.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57411 from akshatshenoi-db/archive-7z.
Authored-by: akshatshenoi-db <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 0b1a92fa82bc574e77a58d3ed25f375d8cc9a7f2)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../datasources/AvroArchiveReadBase.scala | 15 ++++
.../datasources/AvroTarArchiveReadSuite.scala | 28 -------
.../datasources/AvroZipArchiveReadSuite.scala | 28 -------
sql/core/pom.xml | 5 ++
.../datasources/SupportsArchiveFormat.scala | 98 +++++++++++++++++++++-
.../execution/datasources/CSVArchiveReadBase.scala | 89 ++++++++++++++++++++
.../datasources/CSVHeaderArchiveReadBase.scala | 52 ------------
.../datasources/CSVHeaderTarArchiveReadSuite.scala | 28 -------
.../datasources/CSVHeaderZipArchiveReadSuite.scala | 28 -------
.../datasources/CSVHeaderlessArchiveReadBase.scala | 43 ----------
.../CSVHeaderlessTarArchiveReadSuite.scala | 28 -------
.../CSVHeaderlessZipArchiveReadSuite.scala | 28 -------
.../datasources/JSONArchiveReadBase.scala | 15 ++++
.../datasources/JSONTarArchiveReadSuite.scala | 28 -------
.../datasources/JSONZipArchiveReadSuite.scala | 28 -------
.../datasources/SevenZArchiveReadBase.scala | 71 ++++++++++++++++
.../datasources/SupportsArchiveFormatSuite.scala | 73 ++++++++++++++++
.../datasources/TextArchiveReadBase.scala | 15 ++++
.../datasources/TextTarArchiveReadSuite.scala | 27 ------
.../datasources/TextZipArchiveReadSuite.scala | 27 ------
.../execution/datasources/XMLArchiveReadBase.scala | 15 ++++
.../datasources/XMLTarArchiveReadSuite.scala | 28 -------
.../datasources/XMLZipArchiveReadSuite.scala | 28 -------
23 files changed, 392 insertions(+), 433 deletions(-)
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroArchiveReadBase.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroArchiveReadBase.scala
index 0513cfe7db96..eb64fa2a6e5f 100644
---
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroArchiveReadBase.scala
+++
b/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroArchiveReadBase.scala
@@ -78,3 +78,18 @@ trait AvroArchiveReadBase extends ArchiveReadSuiteBase {
} finally stream.close()
}
}
+
+class AvroTarArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with AvroArchiveReadBase
+ with TarArchiveReadBase
+
+class AvroZipArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with AvroArchiveReadBase
+ with ZipArchiveReadBase
+
+class AvroSevenZArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with AvroArchiveReadBase
+ with SevenZArchiveReadBase
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroTarArchiveReadSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroTarArchiveReadSuite.scala
deleted file mode 100644
index 00aa8396ed95..000000000000
---
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of Avro files packed in tar archives (`.tar`/`.tar.gz`/`.tgz`): the
shared archive tests
- * from [[ArchiveReadSuiteBase]] plus the Avro-specific ones from
[[AvroArchiveReadBase]], run over
- * tar containers via [[TarArchiveReadBase]].
- */
-class AvroTarArchiveReadSuite
- extends ArchiveReadSuiteBase
- with AvroArchiveReadBase
- with TarArchiveReadBase
diff --git
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroZipArchiveReadSuite.scala
b/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroZipArchiveReadSuite.scala
deleted file mode 100644
index 24281fb6637f..000000000000
---
a/connector/avro/src/test/scala/org/apache/spark/sql/execution/datasources/AvroZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of Avro files packed in zip archives (`.zip`): the shared archive
tests from
- * [[ArchiveReadSuiteBase]] plus the Avro-specific ones from
[[AvroArchiveReadBase]], run over zip
- * containers via [[ZipArchiveReadBase]].
- */
-class AvroZipArchiveReadSuite
- extends ArchiveReadSuiteBase
- with AvroArchiveReadBase
- with ZipArchiveReadBase
diff --git a/sql/core/pom.xml b/sql/core/pom.xml
index b545b516b05e..bce2e5e2d929 100644
--- a/sql/core/pom.xml
+++ b/sql/core/pom.xml
@@ -296,6 +296,11 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</dependency>
+ <!-- commons-compress reads 7z archives but treats xz (LZMA/LZMA2) as an
optional dependency. -->
+ <dependency>
+ <groupId>org.tukaani</groupId>
+ <artifactId>xz</artifactId>
+ </dependency>
<dependency>
<groupId>org.bouncycastle</groupId>
<artifactId>bcpkix-jdk18on</artifactId>
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
index 105bad108967..ffb38c52d9c5 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
@@ -18,6 +18,8 @@
package org.apache.spark.sql.execution.datasources
import java.io.{Closeable, InputStream}
+import java.nio.ByteBuffer
+import java.nio.channels.{NonWritableChannelException, SeekableByteChannel}
import java.util.Locale
import java.util.regex.Pattern
import java.util.zip.GZIPInputStream
@@ -25,12 +27,13 @@ import java.util.zip.GZIPInputStream
import scala.util.control.NonFatal
import org.apache.commons.compress.archivers.{ArchiveEntry, ArchiveInputStream}
+import org.apache.commons.compress.archivers.sevenz.{SevenZArchiveEntry,
SevenZFile}
import org.apache.commons.compress.archivers.tar.TarArchiveInputStream
import org.apache.commons.compress.archivers.zip.ZipArchiveInputStream
import org.apache.commons.io.ByteOrderMark
import org.apache.commons.io.input.{BOMInputStream, CloseShieldInputStream}
import org.apache.hadoop.conf.Configuration
-import org.apache.hadoop.fs.Path
+import org.apache.hadoop.fs.{FSDataInputStream, Path}
import org.apache.hadoop.io.Text
import org.apache.hadoop.util.LineReader
@@ -39,7 +42,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.util.HadoopFSUtils
/**
- * Mixed into a file-based data source to let it read tar/zip archives of its
files, treating an
+ * Mixed into a file-based data source to let it read tar/zip/7z archives of
its files, treating an
* archive like a directory of the entries it contains. Schema inference is
left to each format.
*/
trait SupportsArchiveFormat extends Logging {
@@ -98,7 +101,7 @@ object SupportsArchiveFormat {
def isArchivePath(path: Path): Boolean = {
val name = path.getName.toLowerCase(Locale.ROOT)
name.endsWith(".tar") || name.endsWith(".tar.gz") || name.endsWith(".tgz")
||
- name.endsWith(".zip")
+ name.endsWith(".zip") || name.endsWith(".7z")
}
/**
@@ -123,9 +126,34 @@ object SupportsArchiveFormat {
}
case n if n.endsWith(".zip") =>
new
ZipArchiveInputStream(CodecStreams.createInputStreamWithCloseResource(conf,
path))
+ case n if n.endsWith(".7z") =>
+ openSevenZStream(path, conf)
case _ =>
throw new IllegalArgumentException(
- s"$path is not a supported archive (expected .tar, .tar.gz, .tgz, or
.zip)")
+ s"$path is not a supported archive (expected .tar, .tar.gz, .tgz,
.zip, or .7z)")
+ }
+ }
+
+ /**
+ * Opens a `.7z` archive by seeking.
+ *
+ * @param path the archive path
+ * @param conf Hadoop configuration used to open the archive
+ * @return the archive's entries as an [[ArchiveInputStream]] cursor
+ */
+ private def openSevenZStream(
+ path: Path,
+ conf: Configuration): ArchiveInputStream[_ <: ArchiveEntry] = {
+ val fs = path.getFileSystem(conf)
+ val in = fs.open(path)
+ try {
+ val channel = new HadoopSeekableByteChannel(in,
fs.getFileStatus(path).getLen)
+ new SevenZArchiveInputStream(
+ SevenZFile.builder().setSeekableByteChannel(channel).get(), channel)
+ } catch {
+ case NonFatal(e) =>
+ try in.close() catch { case NonFatal(_) => }
+ throw e
}
}
@@ -231,3 +259,65 @@ object SupportsArchiveFormat {
}
}
+
+/**
+ * Adapter for [[FSDataInputStream]] to [[SeekableByteChannel]].
+ *
+ * @param in the underlying seekable Hadoop stream
+ * @param length the file's size in bytes
+ */
+private class HadoopSeekableByteChannel(in: FSDataInputStream, length: Long)
+ extends SeekableByteChannel {
+
+ private var open = true
+
+ override def read(dst: ByteBuffer): Int = {
+ val len = dst.remaining()
+ if (len == 0) return 0
+ // Read straight into dst's backing array when it exposes one, avoiding a
per-call copy.
+ if (dst.hasArray) {
+ val n = in.read(dst.array(), dst.arrayOffset() + dst.position(), len)
+ if (n > 0) dst.position(dst.position() + n)
+ n
+ } else {
+ val buf = new Array[Byte](len)
+ val n = in.read(buf, 0, len)
+ if (n > 0) dst.put(buf, 0, n)
+ n
+ }
+ }
+
+ override def position(): Long = in.getPos
+ override def position(newPosition: Long): SeekableByteChannel = {
in.seek(newPosition); this }
+ override def size(): Long = length
+ override def isOpen: Boolean = open
+ override def close(): Unit = { open = false; in.close() }
+
+ override def write(src: ByteBuffer): Int = throw new
NonWritableChannelException
+ override def truncate(size: Long): SeekableByteChannel = throw new
NonWritableChannelException
+}
+
+/**
+ * Exposes the current [[SevenZFile]] entry's bytes as a plain forward
[[InputStream]].
+ *
+ * @param sevenZ the 7z file, positioned at the current entry
+ */
+private class SevenZEntryInputStream(sevenZ: SevenZFile) extends InputStream {
+ override def read(): Int = sevenZ.read()
+ override def read(b: Array[Byte], off: Int, len: Int): Int = sevenZ.read(b,
off, len)
+}
+
+/**
+ * Adapts a [[SevenZFile]] to the [[ArchiveInputStream]] cursor the engine
consumes. Closing it
+ * closes both the `SevenZFile` and the channel it reads from, which
`SevenZFile` does not own.
+ *
+ * @param sevenZ the 7z file to adapt
+ * @param channel the channel `sevenZ` reads from, closed alongside it
+ */
+private class SevenZArchiveInputStream(sevenZ: SevenZFile, channel:
SeekableByteChannel)
+ extends ArchiveInputStream[SevenZArchiveEntry](new
SevenZEntryInputStream(sevenZ), "UTF-8") {
+
+ override def getNextEntry(): SevenZArchiveEntry = sevenZ.getNextEntry
+
+ override def close(): Unit = try sevenZ.close() finally channel.close()
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
index 9ad49235b3fb..b3ab43fcd19c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
@@ -150,3 +150,92 @@ trait CSVArchiveReadBase extends ArchiveReadSuiteBase {
}
}
}
+
+/**
+ * [[CSVArchiveReadBase]] reading CSV files that carry a header row, plus
header-specific archive
+ * tests (mismatched headers, and delimiter/multiline cases whose first row is
a header).
+ */
+trait CSVHeaderArchiveReadBase extends CSVArchiveReadBase {
+
+ import testImplicits._
+
+ override protected def header: Boolean = true
+
+ test("CSV: entries with mismatched headers behave like standalone files") {
+ assertArchiveMatchesDir(
+ Seq(
+ entryName(0) -> encodeFile(sampleDf((1, "Alice"), (2, "Bob"))),
+ // A different second-column header: the schema's "name" column is
absent from this entry.
+ entryName(1) -> encodeFile(Seq((3, "Carol")).toDF("id", "nickname"))))
+ }
+
+ test("CSV: custom delimiter matches a directory read") {
+ assertArchiveMatchesDir(
+ Seq("a.csv" -> csvBytes("id;name\n1;Alice\n2;Bob\n")),
+ extraOptions = Map("delimiter" -> ";"))
+ }
+
+ test("CSV: multiline quoted fields with embedded newlines match a directory
read") {
+ assertArchiveMatchesDir(
+ Seq(
+ "a.csv" -> csvBytes("id,note\n1,\"line1\nline2\"\n2,\"plain\"\n"),
+ "b.csv" -> csvBytes("id,note\n3,\"a\nb\nc\"\n")),
+ extraOptions = Map("multiLine" -> "true"),
+ schema = "id INT, note STRING")
+ }
+}
+
+/**
+ * [[CSVArchiveReadBase]] reading headerless CSV files (columns are
positional), plus headerless
+ * delimiter/multiline archive tests. The shared archive tests from
[[ArchiveReadSuiteBase]] cover
+ * the common headerless read paths.
+ */
+trait CSVHeaderlessArchiveReadBase extends CSVArchiveReadBase {
+
+ override protected def header: Boolean = false
+
+ test("CSV: headerless custom delimiter matches a directory read") {
+ assertArchiveMatchesDir(
+ Seq("a.csv" -> csvBytes("1;Alice\n2;Bob\n"), "b.csv" ->
csvBytes("3;Carol\n")),
+ extraOptions = Map("delimiter" -> ";"))
+ }
+
+ test("CSV: headerless multiline quoted fields with embedded newlines match a
directory read") {
+ assertArchiveMatchesDir(
+ Seq(
+ "a.csv" -> csvBytes("1,\"line1\nline2\"\n2,\"plain\"\n"),
+ "b.csv" -> csvBytes("3,\"a\nb\nc\"\n")),
+ extraOptions = Map("multiLine" -> "true"),
+ schema = "id INT, note STRING")
+ }
+}
+
+class CSVHeaderTarArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderArchiveReadBase
+ with TarArchiveReadBase
+
+class CSVHeaderZipArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderArchiveReadBase
+ with ZipArchiveReadBase
+
+class CSVHeaderSevenZArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderArchiveReadBase
+ with SevenZArchiveReadBase
+
+class CSVHeaderlessTarArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderlessArchiveReadBase
+ with TarArchiveReadBase
+
+class CSVHeaderlessZipArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderlessArchiveReadBase
+ with ZipArchiveReadBase
+
+class CSVHeaderlessSevenZArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with CSVHeaderlessArchiveReadBase
+ with SevenZArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderArchiveReadBase.scala
deleted file mode 100644
index 32ad6e818076..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderArchiveReadBase.scala
+++ /dev/null
@@ -1,52 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * [[CSVArchiveReadBase]] reading CSV files that carry a header row, plus
header-specific archive
- * tests (mismatched headers, and delimiter/multiline cases whose first row is
a header).
- */
-trait CSVHeaderArchiveReadBase extends CSVArchiveReadBase {
-
- import testImplicits._
-
- override protected def header: Boolean = true
-
- test("CSV: entries with mismatched headers behave like standalone files") {
- assertArchiveMatchesDir(
- Seq(
- entryName(0) -> encodeFile(sampleDf((1, "Alice"), (2, "Bob"))),
- // A different second-column header: the schema's "name" column is
absent from this entry.
- entryName(1) -> encodeFile(Seq((3, "Carol")).toDF("id", "nickname"))))
- }
-
- test("CSV: custom delimiter matches a directory read") {
- assertArchiveMatchesDir(
- Seq("a.csv" -> csvBytes("id;name\n1;Alice\n2;Bob\n")),
- extraOptions = Map("delimiter" -> ";"))
- }
-
- test("CSV: multiline quoted fields with embedded newlines match a directory
read") {
- assertArchiveMatchesDir(
- Seq(
- "a.csv" -> csvBytes("id,note\n1,\"line1\nline2\"\n2,\"plain\"\n"),
- "b.csv" -> csvBytes("id,note\n3,\"a\nb\nc\"\n")),
- extraOptions = Map("multiLine" -> "true"),
- schema = "id INT, note STRING")
- }
-}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderTarArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderTarArchiveReadSuite.scala
deleted file mode 100644
index 7ac2ad4084aa..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of header-carrying CSV files packed in tar archives
(`.tar`/`.tar.gz`/`.tgz`): the shared
- * archive tests from [[ArchiveReadSuiteBase]] plus the header-mode CSV tests
from
- * [[CSVHeaderArchiveReadBase]], run over tar containers via
[[TarArchiveReadBase]].
- */
-class CSVHeaderTarArchiveReadSuite
- extends ArchiveReadSuiteBase
- with CSVHeaderArchiveReadBase
- with TarArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderZipArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderZipArchiveReadSuite.scala
deleted file mode 100644
index eb3bf1d78596..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of header-carrying CSV files packed in zip archives (`.zip`): the
shared archive tests from
- * [[ArchiveReadSuiteBase]] plus the header-mode CSV tests from
[[CSVHeaderArchiveReadBase]], run
- * over zip containers via [[ZipArchiveReadBase]].
- */
-class CSVHeaderZipArchiveReadSuite
- extends ArchiveReadSuiteBase
- with CSVHeaderArchiveReadBase
- with ZipArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessArchiveReadBase.scala
deleted file mode 100644
index ba4d7f63bb46..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessArchiveReadBase.scala
+++ /dev/null
@@ -1,43 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * [[CSVArchiveReadBase]] reading headerless CSV files (columns are
positional), plus headerless
- * delimiter/multiline archive tests. The shared archive tests from
[[ArchiveReadSuiteBase]] cover
- * the common headerless read paths.
- */
-trait CSVHeaderlessArchiveReadBase extends CSVArchiveReadBase {
-
- override protected def header: Boolean = false
-
- test("CSV: headerless custom delimiter matches a directory read") {
- assertArchiveMatchesDir(
- Seq("a.csv" -> csvBytes("1;Alice\n2;Bob\n"), "b.csv" ->
csvBytes("3;Carol\n")),
- extraOptions = Map("delimiter" -> ";"))
- }
-
- test("CSV: headerless multiline quoted fields with embedded newlines match a
directory read") {
- assertArchiveMatchesDir(
- Seq(
- "a.csv" -> csvBytes("1,\"line1\nline2\"\n2,\"plain\"\n"),
- "b.csv" -> csvBytes("3,\"a\nb\nc\"\n")),
- extraOptions = Map("multiLine" -> "true"),
- schema = "id INT, note STRING")
- }
-}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessTarArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessTarArchiveReadSuite.scala
deleted file mode 100644
index 28316c97ca8c..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of headerless CSV files packed in tar archives
(`.tar`/`.tar.gz`/`.tgz`): the shared
- * archive tests from [[ArchiveReadSuiteBase]] plus the headerless CSV tests
from
- * [[CSVHeaderlessArchiveReadBase]], run over tar containers via
[[TarArchiveReadBase]].
- */
-class CSVHeaderlessTarArchiveReadSuite
- extends ArchiveReadSuiteBase
- with CSVHeaderlessArchiveReadBase
- with TarArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessZipArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessZipArchiveReadSuite.scala
deleted file mode 100644
index b4bddb603725..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVHeaderlessZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of headerless CSV files packed in zip archives (`.zip`): the shared
archive tests from
- * [[ArchiveReadSuiteBase]] plus the headerless CSV tests from
[[CSVHeaderlessArchiveReadBase]], run
- * over zip containers via [[ZipArchiveReadBase]].
- */
-class CSVHeaderlessZipArchiveReadSuite
- extends ArchiveReadSuiteBase
- with CSVHeaderlessArchiveReadBase
- with ZipArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
index 51e89832a4e5..24a187380535 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
@@ -202,3 +202,18 @@ trait JSONArchiveReadBase extends ArchiveReadSuiteBase {
}
}
}
+
+class JSONTarArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with JSONArchiveReadBase
+ with TarArchiveReadBase
+
+class JSONZipArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with JSONArchiveReadBase
+ with ZipArchiveReadBase
+
+class JSONSevenZArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with JSONArchiveReadBase
+ with SevenZArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONTarArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONTarArchiveReadSuite.scala
deleted file mode 100644
index 5a703a321c59..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of JSON files packed in tar archives (`.tar`/`.tar.gz`/`.tgz`): the
shared archive tests
- * from [[ArchiveReadSuiteBase]] plus the JSON-specific ones from
[[JSONArchiveReadBase]], run over
- * tar containers via [[TarArchiveReadBase]].
- */
-class JSONTarArchiveReadSuite
- extends ArchiveReadSuiteBase
- with JSONArchiveReadBase
- with TarArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONZipArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONZipArchiveReadSuite.scala
deleted file mode 100644
index 76cee32238b3..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of JSON files packed in zip archives (`.zip`): the shared archive
tests from
- * [[ArchiveReadSuiteBase]] plus the JSON-specific ones from
[[JSONArchiveReadBase]], run over zip
- * containers via [[ZipArchiveReadBase]].
- */
-class JSONZipArchiveReadSuite
- extends ArchiveReadSuiteBase
- with JSONArchiveReadBase
- with ZipArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SevenZArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SevenZArchiveReadBase.scala
new file mode 100644
index 000000000000..efc6c280bfb6
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SevenZArchiveReadBase.scala
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.Files
+
+import org.apache.commons.compress.archivers.sevenz.{SevenZArchiveEntry,
SevenZOutputFile}
+
+/**
+ * Binds [[ArchiveReadSuiteBase]]'s archive-format hooks to 7z containers
(`.7z`). The
+ * container-writing helpers live in [[SevenZArchiveTestUtils]] (shared with
standalone suites that
+ * cannot extend `ArchiveReadSuiteBase`). Reusable across file formats -- a
+ * `<format>SevenZArchiveReadSuite` mixes this in alongside the file-format
trait.
+ */
+trait SevenZArchiveReadBase extends ArchiveReadSuiteBase with
SevenZArchiveTestUtils {
+
+ override protected def corruptArchiveExtension: String = "7z"
+}
+
+/**
+ * 7z container-writing helpers (`.7z`), independent of any read-test harness
-- the 7z peer of
+ * [[TarArchiveTestUtils]] and [[ZipArchiveTestUtils]].
[[SevenZArchiveReadBase]] mixes this into
+ * [[ArchiveReadSuiteBase]] for the format-agnostic suites, and standalone
suites that cannot extend
+ * `ArchiveReadSuiteBase` (e.g. `TextSevenZArchiveReadSuite`) mix it in
directly, so the container
+ * logic lives in one place.
+ */
+trait SevenZArchiveTestUtils {
+
+ /** 7z extensions to exercise; the head is the default. */
+ protected def archiveExtensions: Seq[String] = Seq("7z")
+
+ /** Writes `entries` (name -> bytes) into the 7z archive at `dest` (default
LZMA2 compression). */
+ protected def writeArchive(dest: File, entries: Seq[(String, Array[Byte])]):
Unit = {
+ val out = new SevenZOutputFile(dest)
+ try {
+ entries.foreach { case (entryName, bytes) =>
+ val entry = new SevenZArchiveEntry()
+ entry.setName(entryName)
+ out.putArchiveEntry(entry)
+ out.write(bytes)
+ out.closeArchiveEntry()
+ }
+ out.finish()
+ } finally out.close()
+ }
+
+ /**
+ * Writes bytes that are not a readable 7z archive to `dest`. The leading
bytes are not the 7z
+ * signature, so `SevenZFile` fails when the archive is opened (rather than
reporting it empty).
+ */
+ protected def writeCorruptArchive(dest: File): Unit =
+ Files.write(dest.toPath, "this is not a valid 7z archive, just some random
bytes"
+ .getBytes(StandardCharsets.UTF_8))
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
index 65547991c6b2..4247e82a1097 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
@@ -25,6 +25,7 @@ import java.util.zip.GZIPOutputStream
import scala.collection.mutable.ArrayBuffer
+import org.apache.commons.compress.archivers.sevenz.{SevenZArchiveEntry,
SevenZOutputFile}
import org.apache.commons.compress.archivers.tar.{TarArchiveEntry,
TarArchiveOutputStream}
import org.apache.commons.compress.archivers.zip.{ZipArchiveEntry,
ZipArchiveOutputStream}
import org.apache.hadoop.conf.Configuration
@@ -122,6 +123,24 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
try fos.write(out.toByteArray) finally fos.close()
}
+ /** Write a 7z archive, used to verify the `.7z` archive path. */
+ private def writeSevenZ(file: File, entries: Seq[Entry]): Unit = {
+ val out = new SevenZOutputFile(file)
+ try {
+ entries.foreach { e =>
+ // 7z records an explicit directory flag rather than inferring it from
a trailing slash.
+ val sevenZEntry = new SevenZArchiveEntry()
+ sevenZEntry.setName(e.name)
+ sevenZEntry.setDirectory(e.isDir)
+ sevenZEntry.setHasStream(!e.isDir)
+ out.putArchiveEntry(sevenZEntry)
+ if (!e.isDir) out.write(e.data)
+ out.closeArchiveEntry()
+ }
+ out.finish()
+ } finally out.close()
+ }
+
private def textEntry(name: String, body: String): Entry =
Entry(name, body.getBytes(StandardCharsets.UTF_8))
@@ -445,4 +464,58 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
s"expected a clear unsupported-feature error, got $ex")
}
}
+
+ // ----- 7z -----------------------------------------------------------------
+ // 7z keeps its entry index at the end of the file, so the reader seeks
rather than streaming
+ // forward like tar and zip. These cases confirm the `.7z` dispatch and that
the seek-based
+ // container surfaces entries through the same engine.
+
+ test("readArchiveEntries: empty 7z yields empty iterator") {
+ withTempDir { dir =>
+ val sevenZ = new File(dir, "empty.7z")
+ writeSevenZ(sevenZ, Seq.empty)
+ assert(collect(sevenZ).isEmpty)
+ }
+ }
+
+ test("readArchiveEntries: 7z single entry exposes its name and bytes") {
+ withTempDir { dir =>
+ val sevenZ = new File(dir, "single.7z")
+ writeSevenZ(sevenZ, Seq(textEntry("only.csv", "hello\n")))
+ assert(collect(sevenZ) == Seq("only.csv" -> "hello\n"))
+ }
+ }
+
+ test("readArchiveEntries: 7z multiple entries chained in archive order") {
+ withTempDir { dir =>
+ val sevenZ = new File(dir, "multi.7z")
+ writeSevenZ(sevenZ,
+ Seq(textEntry("a.csv", "a"), textEntry("b.csv", "b"),
textEntry("c.csv", "c")))
+ assert(collect(sevenZ) == Seq("a.csv" -> "a", "b.csv" -> "b", "c.csv" ->
"c"))
+ }
+ }
+
+ test("readArchiveEntries: 7z directory entries are skipped") {
+ withTempDir { dir =>
+ val sevenZ = new File(dir, "dirs.7z")
+ writeSevenZ(sevenZ, Seq(
+ Entry("subdir", Array.emptyByteArray, isDir = true),
+ textEntry("subdir/data.csv", "x")))
+ assert(collect(sevenZ) == Seq("subdir/data.csv" -> "x"))
+ }
+ }
+
+ test("readArchiveEntries: 7z dotfile, underscore-marker, and prefixed-dir
entries are skipped") {
+ withTempDir { dir =>
+ val sevenZ = new File(dir, "skipped.7z")
+ writeSevenZ(sevenZ, Seq(
+ textEntry("._real.csv", "junk"), // macOS AppleDouble sidecar
+ textEntry(".hidden", "ignored"), // bare dotfile
+ textEntry("_SUCCESS", "marker"), // _-prefixed marker
(InMemoryFileIndex skips it)
+ textEntry("_temporary/part-0.csv", "tmp"), // entry under a _-prefixed
dir (skipped whole)
+ textEntry("real.csv", "kept"),
+ textEntry("nested/._sidecar", "junk2"))) // dotfile in a subdir
+ assert(collect(sevenZ) == Seq("real.csv" -> "kept"))
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
index ba0cc64e1b35..1204c5a71b04 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
@@ -189,3 +189,18 @@ trait TextArchiveReadBase extends QueryTest with
SharedSparkSession {
}
}
}
+
+class TextTarArchiveReadSuite extends TextArchiveReadBase with
TarArchiveTestUtils {
+
+ override protected def corruptArchiveExtension: String = "tar.gz"
+}
+
+class TextZipArchiveReadSuite extends TextArchiveReadBase with
ZipArchiveTestUtils {
+
+ override protected def corruptArchiveExtension: String = "zip"
+}
+
+class TextSevenZArchiveReadSuite extends TextArchiveReadBase with
SevenZArchiveTestUtils {
+
+ override protected def corruptArchiveExtension: String = "7z"
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextTarArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextTarArchiveReadSuite.scala
deleted file mode 100644
index 1764bfdefcfb..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of text files packed in tar archives (`.tar`/`.tar.gz`/`.tgz`): the
shared text archive
- * tests from [[TextArchiveReadBase]], run over tar containers via
[[TarArchiveTestUtils]].
- */
-class TextTarArchiveReadSuite extends TextArchiveReadBase with
TarArchiveTestUtils {
-
- override protected def corruptArchiveExtension: String = "tar.gz"
-}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextZipArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextZipArchiveReadSuite.scala
deleted file mode 100644
index a22951e8d5df..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,27 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of text files packed in zip archives (`.zip`): the shared text
archive tests from
- * [[TextArchiveReadBase]], run over zip containers via
[[ZipArchiveTestUtils]].
- */
-class TextZipArchiveReadSuite extends TextArchiveReadBase with
ZipArchiveTestUtils {
-
- override protected def corruptArchiveExtension: String = "zip"
-}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
index 71f97dd32637..c5e956d4cd38 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
@@ -133,3 +133,18 @@ trait XMLArchiveReadBase extends ArchiveReadSuiteBase {
schema = corruptSchema)
}
}
+
+class XMLTarArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with XMLArchiveReadBase
+ with TarArchiveReadBase
+
+class XMLZipArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with XMLArchiveReadBase
+ with ZipArchiveReadBase
+
+class XMLSevenZArchiveReadSuite
+ extends ArchiveReadSuiteBase
+ with XMLArchiveReadBase
+ with SevenZArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLTarArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLTarArchiveReadSuite.scala
deleted file mode 100644
index ec4a7047d600..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLTarArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of XML files packed in tar archives (`.tar`/`.tar.gz`/`.tgz`): the
shared archive tests
- * from [[ArchiveReadSuiteBase]] plus the XML-specific ones from
[[XMLArchiveReadBase]], run over
- * tar containers via [[TarArchiveReadBase]].
- */
-class XMLTarArchiveReadSuite
- extends ArchiveReadSuiteBase
- with XMLArchiveReadBase
- with TarArchiveReadBase
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLZipArchiveReadSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLZipArchiveReadSuite.scala
deleted file mode 100644
index 9721561b756f..000000000000
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLZipArchiveReadSuite.scala
+++ /dev/null
@@ -1,28 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.spark.sql.execution.datasources
-
-/**
- * Reads of XML files packed in zip archives (`.zip`): the shared archive
tests from
- * [[ArchiveReadSuiteBase]] plus the XML-specific ones from
[[XMLArchiveReadBase]], run over zip
- * containers via [[ZipArchiveReadBase]].
- */
-class XMLZipArchiveReadSuite
- extends ArchiveReadSuiteBase
- with XMLArchiveReadBase
- with ZipArchiveReadBase
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
