This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 73ca206be32a [SPARK-60217][SQL] Add a decompression read option to
text-based file sources
73ca206be32a is described below
commit 73ca206be32a4b7e8d5a151728f68c5c2a4f396e
Author: Sandip Agarwala <[email protected]>
AuthorDate: Wed Nov 4 07:10:19 2026 +0900
[SPARK-60217][SQL] Add a decompression read option to text-based file
sources
### What changes were proposed in this pull request?
This PR adds a read-only data source option, `decompression`, to the CSV,
JSON, XML and text file sources. When set, it forces the codec used to
decompress every input file instead of inferring the codec from each file name
extension:
```scala
spark.read.option("decompression", "gzip").csv("/path/to/data.csv")
```
- Accepts the short names supported by the `compression` write option
(`none`, `uncompressed`, `bzip2`, `deflate`, `gzip`, `lz4`, `snappy`), plus
`zstd`/`zstandard`/`zst`, or a fully qualified Hadoop `CompressionCodec` class
name. Names are case-insensitive.
- `none`/`uncompressed` reads the files as-is, even when the file name
carries a compression extension.
How it works:
- `HadoopCodecStreams.getDecompressionCodec` is the single place every
text-based reader already uses to pick a codec. It now honors an internal
Hadoop conf key
(`org.apache.spark.sql.execution.datasources.decompressionCodec`) before
falling back to extension-based detection.
- `CompressionCodecs.setReadCodecConfiguration` puts the parsed option into
the per-scan Hadoop conf before it is broadcast. It is called once for DSv1, in
`TextBasedFileFormat.buildReaderWithPartitionValues`, and once for DSv2, in
`TextBasedFileScan.newReadHadoopConf`, plus in the multiLine schema-inference
paths. Line-based inference reuses `TextFileFormat`, which forwards the
options. When the option is absent, the key is explicitly unset, so a value
inherited from the session or H [...]
- Splittability follows the forced codec: a forced bzip2 file stays
splittable, a forced gzip file does not, and `none` keeps the file splittable.
For DSv1 this is decided in `TextBasedFileFormat.isSplitable`, so splitting
always agrees with decompression for every `TextBasedFileFormat`, including
libsvm. V2 scans report the forced codec as the reason a file is not split.
- Spark's own zstd codec, used when Hadoop's cannot create a stream, is
applied only when `zstd` is the forced codec, or when no codec is forced and
the name has a zstd extension; it never replaces another forced codec.
- The option is parsed once in `FileSourceOptions`. The name lives in
`DataSourceOptions` and is registered by the four supporting formats.
Not supported. Both fail with a clear `UNSUPPORTED_FEATURE` error on the
driver while planning or inferring the schema, so `ignoreCorruptFiles` cannot
hide them:
- archive inputs (`.tar`, `.tar.gz`, `.tgz`, `.zip`, `.7z`) when archive
reading is enabled (`spark.sql.files.archive.reader.enabled`), since the
container format is determined by the extension
(`UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE`);
- the legacy Hadoop `LineRecordReader`
(`spark.sql.execution.datasources.hadoopLineRecordReader.enabled=false`), which
cannot be told which codec to use
(`UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_WITH_LEGACY_LINE_READER`).
### Why are the changes needed?
Text-based file sources choose the decompression codec solely from the file
name extension. Files whose extension does not match their actual compression
cannot be read correctly:
- a gzip-compressed file named `data.csv` is read as garbage;
- a plain-text file named `data.csv.gz` fails to decompress.
Writers already have a `compression` option, but readers had no way to
override extension-based detection.
A separate option name is used on purpose instead of honoring `compression`
on read. `compression` is often persisted in table options, for example by
`saveAsTable` or `CREATE TABLE ... OPTIONS (compression 'gzip')`, or shared
between read and write option maps. Honoring it on read would change how
existing tables and pipelines read their files.
### Does this PR introduce _any_ user-facing change?
Yes. There is a new read option, `decompression`, for the CSV, JSON, XML
and text data sources, documented in
`docs/sql-data-sources-{csv,json,text,xml}.md`. Existing reads are unchanged;
in particular, `compression` is still ignored on read.
### How was this patch tested?
New tests in `CSVSuite`, `JsonSuite`, `XmlSuite` and `TextSuite`, which run
on DSv1 and DSv2 where applicable:
- every codec, including zstd, read from a file whose name has no codec
extension, with and without multiLine/wholetext and schema inference;
- `none`/`uncompressed` on a plain file with a `.gz` name;
- codec class names and case-insensitive short names (CSV);
- table options in `CREATE TABLE ... USING csv OPTIONS (...)` and
Structured Streaming reads (CSV);
- splitting with forced bzip2, gzip and `none`, and the interaction with
multiLine (CSV, JSON, text) and libsvm (`LibSVMRelationSuite`);
- the V2 unsplittable reason (`CSVv2Suite`, `TextV2Suite`);
- `compression` alone still does not force a codec on read, and an
inherited internal conf key is ignored;
- invalid codec names, and the legacy line reader error.
Shared tests in `ArchiveReadSuiteBase` and `TextArchiveReadBase` check that
`decompression` is rejected for every archive type and format, including with
`ignoreCorruptFiles=true`, during schema inference, and for archives mixed with
loose files.
Ran locally:
```
build/sbt 'core/testOnly *SparkThrowableSuite'
build/sbt 'mllib/testOnly *LibSVMRelationSuite'
build/sbt 'sql/testOnly *TextV1Suite *TextV2Suite *CSVv1Suite *CSVv2Suite
*JsonV1Suite *JsonV2Suite *XmlSuite *ArchiveReadSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59435 from sandip-db/compression-option.
Authored-by: Sandip Agarwala <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../src/main/resources/error/error-conditions.json | 10 +
.../org/apache/spark/io/HadoopCodecStreams.scala | 86 +++++++-
docs/sql-data-sources-csv.md | 6 +
docs/sql-data-sources-json.md | 6 +
docs/sql-data-sources-text.md | 6 +
docs/sql-data-sources-xml.md | 6 +
.../ml/source/libsvm/LibSVMRelationSuite.scala | 37 +++-
project/MimaExcludes.scala | 4 +
.../spark/sql/catalyst/DataSourceOptions.scala | 5 +
.../spark/sql/catalyst/FileSourceOptions.scala | 27 ++-
.../apache/spark/sql/catalyst/csv/CSVOptions.scala | 1 +
.../spark/sql/catalyst/json/JSONOptions.scala | 1 +
.../sql/catalyst/util/CompressionCodecs.scala | 69 +++++-
.../apache/spark/sql/catalyst/xml/XmlOptions.scala | 1 +
.../spark/sql/errors/QueryExecutionErrors.scala | 15 ++
.../datasources/HadoopLineRecordReader.java | 2 +-
.../sql/execution/datasources/FileFormat.scala | 38 +++-
.../datasources/SupportsArchiveFormat.scala | 32 ++-
.../execution/datasources/csv/CSVDataSource.scala | 6 +
.../execution/datasources/csv/CSVFileFormat.scala | 1 +
.../datasources/json/JsonDataSource.scala | 7 +-
.../datasources/json/JsonFileFormat.scala | 1 +
.../datasources/text/TextFileFormat.scala | 1 +
.../execution/datasources/text/TextOptions.scala | 1 +
.../datasources/v2/TextBasedFileScan.scala | 29 ++-
.../sql/execution/datasources/v2/csv/CSVScan.scala | 4 +-
.../execution/datasources/v2/json/JsonScan.scala | 4 +-
.../execution/datasources/v2/text/TextScan.scala | 6 +-
.../execution/datasources/xml/XmlDataSource.scala | 7 +-
.../execution/datasources/xml/XmlFileFormat.scala | 1 +
.../datasources/ArchiveReadSuiteBase.scala | 76 ++++++-
.../execution/datasources/CSVArchiveReadBase.scala | 2 +
.../datasources/JSONArchiveReadBase.scala | 2 +
.../datasources/SupportsArchiveFormatSuite.scala | 33 ++-
.../datasources/TextArchiveReadBase.scala | 24 +-
.../execution/datasources/XMLArchiveReadBase.scala | 2 +
.../sql/execution/datasources/csv/CSVSuite.scala | 218 ++++++++++++++++++-
.../sql/execution/datasources/json/JsonSuite.scala | 144 +++++++++++-
.../sql/execution/datasources/text/TextSuite.scala | 242 ++++++++++++++++++++-
.../sql/execution/datasources/xml/XmlSuite.scala | 96 +++++++-
40 files changed, 1210 insertions(+), 49 deletions(-)
diff --git a/common/utils/src/main/resources/error/error-conditions.json
b/common/utils/src/main/resources/error/error-conditions.json
index 443d3ebb6f13..9f4e85a8ea44 100644
--- a/common/utils/src/main/resources/error/error-conditions.json
+++ b/common/utils/src/main/resources/error/error-conditions.json
@@ -9306,6 +9306,16 @@
"Direct execution of a CREATE ... <pipelineDatasetType> query. To
create a pipeline dataset, create and execute a pipeline with a SQL file
containing your query attached in the pipeline definition."
]
},
+ "DECOMPRESSION_CODEC_FOR_ARCHIVE" : {
+ "message" : [
+ "The <option> option for the archive file <path>. Archives are
always decompressed based on their file name extension. Remove the option, or
read the archive files separately from the other files."
+ ]
+ },
+ "DECOMPRESSION_CODEC_WITH_LEGACY_LINE_READER" : {
+ "message" : [
+ "The <option> option when <config> is false, because Hadoop's line
reader infers the codec from the file name extension. Set <config> to true, or
remove the option."
+ ]
+ },
"DESC_TABLE_COLUMN_JSON" : {
"message" : [
"DESC TABLE COLUMN AS JSON not supported for individual columns."
diff --git a/core/src/main/scala/org/apache/spark/io/HadoopCodecStreams.scala
b/core/src/main/scala/org/apache/spark/io/HadoopCodecStreams.scala
index cba0e7ffd73b..3f955ffa0fb1 100644
--- a/core/src/main/scala/org/apache/spark/io/HadoopCodecStreams.scala
+++ b/core/src/main/scala/org/apache/spark/io/HadoopCodecStreams.scala
@@ -21,10 +21,12 @@ import java.io.InputStream
import java.util.Locale
import scala.collection.Seq
+import scala.util.control.NonFatal
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.compress._
+import org.apache.hadoop.util.ReflectionUtils
import org.apache.spark.{SparkConf, SparkEnv}
import org.apache.spark.io.{CompressionCodec => SparkCompressionCodec}
@@ -38,8 +40,49 @@ import org.apache.spark.io.{CompressionCodec =>
SparkCompressionCodec}
object HadoopCodecStreams {
private val ZSTD_EXTENSIONS = Seq(".zstd", ".zst")
- // get codec based on file name extension
+ /**
+ * Hadoop `Configuration` key carrying the read-side compression codec
explicitly requested via
+ * the `decompression` data source option. The value is a full codec class
name, an
+ * empty string to force "no compression" (read the file as-is), or absent
to infer the codec
+ * from the file name extension. Set by the file format / scan read paths
and consumed by
+ * [[getDecompressionCodec]] so it uniformly overrides extension-based
detection for line-based,
+ * multi-line, whole-text, and schema-inference reads.
+ *
+ * This is an internal key, deliberately outside the `spark.*` namespace so
it is not mistaken for
+ * a user-facing configuration. The read paths always overwrite or unset it,
so a value inherited
+ * from the session or Hadoop configuration is never honored.
+ */
+ val DECOMPRESSION_CODEC_CONF =
"org.apache.spark.sql.execution.datasources.decompressionCodec"
+
+ /**
+ * Returns the codec to decompress `file` with. When a read compression
option has been set (see
+ * [[DECOMPRESSION_CODEC_CONF]]) it takes precedence over the file name
extension: a codec class
+ * name selects that codec, and an empty value means the file is read as-is.
Otherwise the codec
+ * is inferred from the file name extension.
+ */
def getDecompressionCodec(
+ config: Configuration,
+ file: Path): Option[CompressionCodec] = {
+ Option(config.get(DECOMPRESSION_CODEC_CONF)) match {
+ case Some("") =>
+ // The read option was explicitly set to `none`/`uncompressed`, which
is stored as an empty
+ // string: read the file as-is, ignoring any compression-related file
name extension.
+ None
+ case Some(codecClassName) =>
+ // The compression codec was explicitly specified via the read option,
so use it instead of
+ // inferring from the file name extension. Resolve it the same way the
write path does
+ // (see CodecStreams.getCompressionCodec): load the class and
instantiate it, so any
+ // loadable CompressionCodec is supported, not just those registered
with the factory.
+ val codecClass = config.getClassByName(codecClassName)
+ Some(ReflectionUtils.newInstance(codecClass,
config).asInstanceOf[CompressionCodec])
+ case None =>
+ // No explicit compression option: infer the codec from the file name
extension.
+ getDecompressionCodecFromExtension(config, file)
+ }
+ }
+
+ // get codec based on file name extension
+ private def getDecompressionCodecFromExtension(
config: Configuration,
file: Path): Option[CompressionCodec] = {
val factory = new CompressionCodecFactory(config)
@@ -56,12 +99,23 @@ object HadoopCodecStreams {
}
def createZstdInputStream(
+ config: Configuration,
file: Path,
inputStream: InputStream): Option[InputStream] = {
val sparkConf = Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
val fileName = file.getName.toLowerCase(Locale.ROOT)
- val isOpt = if (ZSTD_EXTENSIONS.exists(fileName.endsWith)) {
+ // Use Spark's Zstandard codec when the read option explicitly requested a
Zstandard codec (in
+ // which case the extension may not match), or, when no codec was forced,
when the file name has
+ // a Zstandard extension. A forced non-Zstandard codec is never replaced
based on the extension.
+ // The conf always holds a resolved codec class name: short names such as
`zstd`, `zstandard`
+ // and `zst` are mapped to `ZStandardCodec` when the option is parsed (see
+ // `CompressionCodecs.getReadCodecClassName`), so comparing class names
covers all of them.
+ val useZstd = Option(config.get(DECOMPRESSION_CODEC_CONF)) match {
+ case Some(codecClassName) => codecClassName ==
classOf[ZStandardCodec].getName
+ case None => ZSTD_EXTENSIONS.exists(fileName.endsWith)
+ }
+ val isOpt = if (useZstd) {
Some(
SparkCompressionCodec
.createCodec(sparkConf, SparkCompressionCodec.ZSTD)
@@ -79,16 +133,28 @@ object HadoopCodecStreams {
val fs = file.getFileSystem(config)
val inputStream: InputStream = fs.open(file)
- getDecompressionCodec(config, file)
- .map { codec =>
+ try {
+ getDecompressionCodec(config, file)
+ .map { codec =>
+ try {
+ codec.createInputStream(inputStream)
+ } catch {
+ case e: RuntimeException =>
+ // createInputStream may fail for ZSTD if hadoop is not already
compiled with ZSTD
+ // support. In that case, we try to use Spark's Zstandard codec.
+ createZstdInputStream(config, file, inputStream).getOrElse(throw
e)
+ }
+ }.getOrElse(inputStream)
+ } catch {
+ case NonFatal(e) =>
+ // No stream wraps the file stream yet, so close it rather than leak
it. This covers a
+ // forced codec that cannot be loaded or created as well as a decoder
that fails.
try {
- codec.createInputStream(inputStream)
+ inputStream.close()
} catch {
- case e: RuntimeException =>
- // createInputStream may fail for ZSTD if hadoop is not already
compiled with ZSTD
- // support. In that case, we try to use Spark's Zstandard codec.
- createZstdInputStream(file, inputStream).getOrElse(throw e)
+ case NonFatal(closeError) => e.addSuppressed(closeError)
}
- }.getOrElse(inputStream)
+ throw e
+ }
}
}
diff --git a/docs/sql-data-sources-csv.md b/docs/sql-data-sources-csv.md
index 7408007e3247..1b2c65d66c95 100644
--- a/docs/sql-data-sources-csv.md
+++ b/docs/sql-data-sources-csv.md
@@ -297,6 +297,12 @@ Data source options of CSV can be set via:
<td>Compression codec to use when saving to file. This can be one of the
known case-insensitive shorten names (<code>none</code>, <code>bzip2</code>,
<code>gzip</code>, <code>lz4</code>, <code>snappy</code> and
<code>deflate</code>). CSV built-in functions ignore this option.</td>
<td>write</td>
</tr>
+ <tr>
+ <td><code>decompression</code></td>
+ <td>(none)</td>
+ <td>Compression codec used to decompress the input files, instead of
inferring it from each file name extension. This can be one of the known
case-insensitive short names (<code>none</code>, <code>uncompressed</code>,
<code>bzip2</code>, <code>deflate</code>, <code>gzip</code>, <code>lz4</code>,
<code>snappy</code>, <code>zstd</code>, <code>zstandard</code> and
<code>zst</code>) or a fully qualified Hadoop <code>CompressionCodec</code>
class name. <code>none</code> and <code>uncompre [...]
+ <td>read</td>
+ </tr>
<tr>
<td><code>timeZone</code></td>
<td>(value of <code>spark.sql.session.timeZone</code> configuration)</td>
diff --git a/docs/sql-data-sources-json.md b/docs/sql-data-sources-json.md
index aadd3c25a780..fca2c83d7fd2 100644
--- a/docs/sql-data-sources-json.md
+++ b/docs/sql-data-sources-json.md
@@ -309,6 +309,12 @@ Data source options of JSON can be set via:
<td>Compression codec to use when saving to file. This can be one of the
known case-insensitive shorten names (none, bzip2, gzip, lz4, snappy and
deflate). JSON built-in functions ignore this option.</td>
<td>write</td>
</tr>
+ <tr>
+ <td><code>decompression</code></td>
+ <td>(none)</td>
+ <td>Compression codec used to decompress the input files, instead of
inferring it from each file name extension. This can be one of the known
case-insensitive short names (<code>none</code>, <code>uncompressed</code>,
<code>bzip2</code>, <code>deflate</code>, <code>gzip</code>, <code>lz4</code>,
<code>snappy</code>, <code>zstd</code>, <code>zstandard</code> and
<code>zst</code>) or a fully qualified Hadoop <code>CompressionCodec</code>
class name. <code>none</code> and <code>uncompre [...]
+ <td>read</td>
+ </tr>
<tr>
<td><code>ignoreNullFields</code></td>
<td>(value of <code>spark.sql.jsonGenerator.ignoreNullFields</code>
configuration)</td>
diff --git a/docs/sql-data-sources-text.md b/docs/sql-data-sources-text.md
index aed8a2e9942f..1469dc16639b 100644
--- a/docs/sql-data-sources-text.md
+++ b/docs/sql-data-sources-text.md
@@ -67,5 +67,11 @@ Data source options of text can be set via:
<td>Compression codec to use when saving to file. This can be one of the
known case-insensitive shorten names (none, bzip2, gzip, lz4, snappy and
deflate).</td>
<td>write</td>
</tr>
+ <tr>
+ <td><code>decompression</code></td>
+ <td>(none)</td>
+ <td>Compression codec used to decompress the input files, instead of
inferring it from each file name extension. This can be one of the known
case-insensitive short names (<code>none</code>, <code>uncompressed</code>,
<code>bzip2</code>, <code>deflate</code>, <code>gzip</code>, <code>lz4</code>,
<code>snappy</code>, <code>zstd</code>, <code>zstandard</code> and
<code>zst</code>) or a fully qualified Hadoop <code>CompressionCodec</code>
class name. <code>none</code> and <code>uncompre [...]
+ <td>read</td>
+ </tr>
</table>
Other generic options can be found in <a
href="https://spark.apache.org/docs/latest/sql-data-sources-generic-options.html">
Generic File Source Options</a>.
diff --git a/docs/sql-data-sources-xml.md b/docs/sql-data-sources-xml.md
index 8a07a486b70d..d1e579663bad 100644
--- a/docs/sql-data-sources-xml.md
+++ b/docs/sql-data-sources-xml.md
@@ -248,6 +248,12 @@ Data source options of XML can be set via:
<td>Compression codec to use when saving to file. This can be one of the
known case-insensitive shortened names (none, bzip2, gzip, lz4, snappy and
deflate). XML built-in functions ignore this option.</td>
<td>write</td>
</tr>
+ <tr>
+ <td><code>decompression</code></td>
+ <td>(none)</td>
+ <td>Compression codec used to decompress the input files, instead of
inferring it from each file name extension. This can be one of the known
case-insensitive short names (<code>none</code>, <code>uncompressed</code>,
<code>bzip2</code>, <code>deflate</code>, <code>gzip</code>, <code>lz4</code>,
<code>snappy</code>, <code>zstd</code>, <code>zstandard</code> and
<code>zst</code>) or a fully qualified Hadoop <code>CompressionCodec</code>
class name. <code>none</code> and <code>uncompre [...]
+ <td>read</td>
+ </tr>
<tr>
<td><code>indent</code></td>
<td>four spaces</td>
diff --git
a/mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
b/mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
index e6650fa58013..3b2f746d6394 100644
---
a/mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/ml/source/libsvm/LibSVMRelationSuite.scala
@@ -17,8 +17,10 @@
package org.apache.spark.ml.source.libsvm
-import java.io.{File, IOException}
+import java.io.{File, FileOutputStream, IOException}
+import java.nio.charset.StandardCharsets
import java.nio.file.Files
+import java.util.zip.GZIPOutputStream
import org.apache.spark.SparkFunSuite
import org.apache.spark.ml.attribute.AttributeGroup
@@ -27,6 +29,7 @@ import org.apache.spark.ml.linalg.SQLDataTypes.VectorType
import org.apache.spark.mllib.util.MLlibTestSparkContext
import org.apache.spark.sql.{Row, SaveMode}
import org.apache.spark.sql.execution.datasources.CommonFileDataSourceSuite
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DoubleType, StructField, StructType}
import org.apache.spark.util.Utils
@@ -223,4 +226,36 @@ class LibSVMRelationSuite
assert(v == Vectors.sparse(2, Seq((0, 2.0), (1, 3.0))))
}
}
+
+ test("SPARK-60217 decompression option decides whether a libsvm file is
split") {
+ withTempDir { dir =>
+ // A gzip-compressed file whose `.libsvm` name says nothing about its
compression.
+ val file = new File(dir, "data.libsvm")
+ val out = new GZIPOutputStream(new FileOutputStream(file))
+ try {
+ out.write((1 to 2000).map(i => s"${i % 2} 1:$i.0").mkString("\n")
+ .getBytes(StandardCharsets.UTF_8))
+ } finally {
+ out.close()
+ }
+ val maxPartitionBytes = SQLConf.FILES_MAX_PARTITION_BYTES.key
+ val previous = spark.conf.getOption(maxPartitionBytes)
+ spark.conf.set(maxPartitionBytes, "256")
+ try {
+ // gzip is not splittable, so the file must be read as a single split,
whether or not
+ // numFeatures has to be inferred first.
+ Seq(Map.empty[String, String], Map("numFeatures" -> "1")).foreach {
options =>
+ val df = spark.read.format("libsvm").options(options)
+ .option("decompression", "gzip").load(file.getCanonicalPath)
+ assert(df.rdd.getNumPartitions == 1, s"options=$options")
+ assert(df.count() == 2000, s"options=$options")
+ }
+ } finally {
+ previous match {
+ case Some(v) => spark.conf.set(maxPartitionBytes, v)
+ case None => spark.conf.unset(maxPartitionBytes)
+ }
+ }
+ }
+ }
}
diff --git a/project/MimaExcludes.scala b/project/MimaExcludes.scala
index eaaa35939f6a..1342c778785d 100644
--- a/project/MimaExcludes.scala
+++ b/project/MimaExcludes.scala
@@ -38,6 +38,10 @@ object MimaExcludes {
// Exclude rules for 4.4.x from 4.3.0 (add 4.4-specific filters below as
needed).
lazy val v44excludes: Seq[Problem => Boolean] = v43excludes ++ Seq(
+ // [SPARK-60217][SQL] Remove the unused createZstdInputStream overload
that built a default
+ // Hadoop conf; callers pass the read's conf to the remaining overload.
+ ProblemFilters.exclude[DirectMissingMethodProblem](
+ "org.apache.spark.io.HadoopCodecStreams.createZstdInputStream"),
// [SPARK-58896] Decision tree leaf counts moved behind NodeStats. The old
package-private
// numLeave accessors are removed from concrete models.
ProblemFilters.exclude[DirectMissingMethodProblem](
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DataSourceOptions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DataSourceOptions.scala
index 0595eef2b8e8..ce8a8ba41be2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DataSourceOptions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/DataSourceOptions.scala
@@ -84,6 +84,11 @@ object DataSourceOptions {
// the corrupt record will be dropped in DROPMALFORMED mode.
val COLUMN_NAME_OF_CORRUPT_RECORD = "columnNameOfCorruptRecord"
+ // The common option name for the text-based file data sources (CSV, JSON,
XML and text) that
+ // support forcing the codec used to decompress the input files, instead of
inferring it from
+ // each file name extension. See `FileSourceOptions.decompressionCodec` for
details.
+ val DECOMPRESSION = "decompression"
+
// The JSON data source option for exploding an embedded top-level array
into rows. See
// `JSONOptions.explodeEmbeddedArray` for details.
val EXPLODE_EMBEDDED_ARRAY = "explodeEmbeddedArray"
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/FileSourceOptions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/FileSourceOptions.scala
index 16593a268803..720d687a42c3 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/FileSourceOptions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/FileSourceOptions.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
import org.apache.hadoop.fs.GlobPattern
import org.apache.spark.sql.catalyst.FileSourceOptions.{ARCHIVE_PATH_FILTER,
IGNORE_CORRUPT_FILES, IGNORE_MISSING_FILES, IGNORED_PATH_SEGMENT_REGEX}
-import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap, DateFormatter}
+import org.apache.spark.sql.catalyst.util.{CaseInsensitiveMap,
CompressionCodecs, DateFormatter}
import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
/**
@@ -99,6 +99,31 @@ class FileSourceOptions(
*/
@transient lazy val archivePathFilterPattern: Option[GlobPattern] =
archivePathFilter.map(FileSourceOptions.compileArchivePathFilter)
+
+ // The raw option value, captured eagerly because `parameters` is transient
and so unavailable
+ // once these options are deserialized on an executor.
+ private val decompressionCodecName: Option[String] =
+ parameters.get(DataSourceOptions.DECOMPRESSION)
+
+ /**
+ * Read-only codec used to decompress the input files instead of inferring
it from their file
+ * name extension, as a codec class name (`null` for `none`/`uncompressed`,
read as-is), or
+ * `None` when the option is absent. Accepts the `compression` short names
plus
+ * `zstd`/`zstandard`/`zst`, which readers support even though writers do
not.
+ *
+ * Honored by the text-based file sources (CSV, JSON, XML and text, which
document it, plus other
+ * `TextBasedFileFormat` sources such as libsvm). Lazy so that the other
file formats, which
+ * ignore the option, never parse or validate it.
+ */
+ lazy val decompressionCodec: Option[String] =
+ decompressionCodecName.map(CompressionCodecs.getReadCodecClassName)
+
+ /**
+ * Whether [[decompressionCodec]] keeps the files splittable (`None` when
the option is absent),
+ * resolved once rather than by loading the codec class for every file.
+ */
+ lazy val decompressionCodecSplittable: Option[Boolean] =
+ decompressionCodec.map(CompressionCodecs.isSplittableReadCodec)
}
object FileSourceOptions {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
index e628192efe4a..a8de46375540 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/csv/CSVOptions.scala
@@ -461,6 +461,7 @@ object CSVOptions extends DataSourceOptions {
val COMPRESSION = "compression"
val CODEC = "codec"
newOption(COMPRESSION, CODEC)
+ val DECOMPRESSION = newOption(DataSourceOptions.DECOMPRESSION)
val SEP = "sep"
val DELIMITER = "delimiter"
newOption(SEP, DELIMITER)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
index 98858076236e..6d2fc97aa7a2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JSONOptions.scala
@@ -327,6 +327,7 @@ object JSONOptions extends DataSourceOptions {
val ALLOW_BACKSLASH_ESCAPING_ANY_CHARACTER =
newOption("allowBackslashEscapingAnyCharacter")
val ALLOW_UNQUOTED_CONTROL_CHARS = newOption("allowUnquotedControlChars")
val COMPRESSION = newOption("compression")
+ val DECOMPRESSION = newOption(DataSourceOptions.DECOMPRESSION)
val MODE = newOption("mode")
val DROP_FIELD_IF_ALL_NULL = newOption("dropFieldIfAllNull")
val IGNORE_NULL_FIELDS = newOption("ignoreNullFields")
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CompressionCodecs.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CompressionCodecs.scala
index 860e73a84598..a46a34b7b39a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CompressionCodecs.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/CompressionCodecs.scala
@@ -21,8 +21,11 @@ import java.util.Locale
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.io.SequenceFile.CompressionType
+import org.apache.hadoop.io.compress.{SplittableCompressionCodec,
ZStandardCodec}
+import org.apache.spark.io.HadoopCodecStreams
import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.util.Utils
object CompressionCodecs {
@@ -30,12 +33,36 @@ object CompressionCodecs {
codec.lowerCaseName() ->
Option(codec.getCompressionCodec).map(_.getClass.getName).orNull
}.toMap
+ // Read-side short names extend the write-side set with Zstandard, which
file-format writers do
+ // not support yet (see the TODO in `HadoopCompressionCodec`) but readers
do, via Spark's
+ // built-in Zstandard codec when Hadoop lacks native support (see
`HadoopCodecStreams`). `zst`,
+ // the most common Zstandard file suffix, is accepted as well.
+ private val readShortCompressionCodecNames = shortCompressionCodecNames ++
Map(
+ "zstd" -> classOf[ZStandardCodec].getName,
+ "zstandard" -> classOf[ZStandardCodec].getName,
+ "zst" -> classOf[ZStandardCodec].getName)
+
/**
* Return the full version of the given codec class.
* If it is already a class name, just return it.
*/
def getCodecClassName(name: String): String = {
- val codecName =
shortCompressionCodecNames.getOrElse(name.toLowerCase(Locale.ROOT), name)
+ getCodecClassName(name, shortCompressionCodecNames)
+ }
+
+ /**
+ * Read-side counterpart of [[getCodecClassName]]. Accepts the same short
names plus `zstd` /
+ * `zstandard`, since the file readers can decompress Zstandard even though
the writers cannot
+ * produce it. Used to parse the `decompression` option.
+ */
+ def getReadCodecClassName(name: String): String = {
+ getCodecClassName(name, readShortCompressionCodecNames)
+ }
+
+ private def getCodecClassName(
+ name: String,
+ shortNames: Map[String, String]): String = {
+ val codecName = shortNames.getOrElse(name.toLowerCase(Locale.ROOT), name)
try {
// Validate the codec name
if (codecName != null) {
@@ -45,7 +72,7 @@ object CompressionCodecs {
} catch {
case _: ClassNotFoundException =>
throw QueryExecutionErrors.codecNotAvailableError(
- codecName, shortCompressionCodecNames.keys.mkString(", "))
+ codecName, shortNames.keys.mkString(", "))
}
}
@@ -66,4 +93,42 @@ object CompressionCodecs {
conf.set("mapreduce.map.output.compress", "false")
}
}
+
+ /**
+ * Records the codec requested via the `decompression` data source option
into the Hadoop
+ * `Configuration`, so the file readers decompress with it instead of
inferring the codec from
+ * the file name extension (see
[[HadoopCodecStreams.getDecompressionCodec]]). Called on the
+ * driver while planning the read, so an unsupported setup fails before any
task runs.
+ *
+ * @param decompressionCodec the parsed `decompression` option: `None` when
absent (unset
+ * the conf so the codec is inferred from the
extension),
+ * `Some(null)` for `none`/`uncompressed` (read
the file as-is), or
+ * `Some(className)` to force that codec.
+ */
+ def setReadCodecConfiguration(
+ conf: Configuration,
+ decompressionCodec: Option[String]): Unit = decompressionCodec match {
+ case Some(codec) =>
+ // Hadoop's `LineRecordReader` infers the codec from the file name
extension and cannot be
+ // told otherwise, so fail rather than silently ignore the option.
+ if (!SQLConf.get.hadoopLineRecordReaderEnabled) {
+ throw
QueryExecutionErrors.decompressionCodecWithLegacyLineReaderError()
+ }
+ // An empty string encodes "no compression" (read as-is); a class name
selects that codec.
+ conf.set(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF,
Option(codec).getOrElse(""))
+ case None =>
+ // Unset rather than leave untouched, so a value inherited from the
session or Hadoop
+ // configuration cannot override the codec of a read that did not ask
for it.
+ conf.unset(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF)
+ }
+
+ /**
+ * Whether files decompressed with the given parsed `decompression` value
can be split into
+ * multiple partitions: `null` (`none`/`uncompressed`) reads the raw file
and stays splittable,
+ * and a codec class is splittable only if it is a
`SplittableCompressionCodec` (e.g. bzip2).
+ */
+ def isSplittableReadCodec(codec: String): Boolean = {
+ codec == null ||
+
classOf[SplittableCompressionCodec].isAssignableFrom(Utils.classForName(codec))
+ }
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala
index 49c82b2c0e56..034907a8e39f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/xml/XmlOptions.scala
@@ -239,6 +239,7 @@ object XmlOptions extends DataSourceOptions {
val MODE = newOption("mode")
val LOCALE = newOption("locale")
val COMPRESSION = newOption("compression")
+ val DECOMPRESSION = newOption(DataSourceOptions.DECOMPRESSION)
val MULTI_LINE = newOption("multiLine")
val SAMPLING_RATIO = newOption("samplingRatio")
val COLUMN_NAME_OF_CORRUPT_RECORD =
newOption(DataSourceOptions.COLUMN_NAME_OF_CORRUPT_RECORD)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
index 9e2dba4d8f59..381bbc634cdf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala
@@ -1018,6 +1018,21 @@ private[sql] object QueryExecutionErrors extends
QueryErrorsBase with ExecutionE
cause = e)
}
+ def decompressionCodecForArchiveUnsupportedError(
+ path: String): SparkUnsupportedOperationException = {
+ new SparkUnsupportedOperationException(
+ errorClass = "UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE",
+ messageParameters = Map("option" -> toDSOption("decompression"), "path"
-> path))
+ }
+
+ def decompressionCodecWithLegacyLineReaderError():
SparkUnsupportedOperationException = {
+ new SparkUnsupportedOperationException(
+ errorClass =
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_WITH_LEGACY_LINE_READER",
+ messageParameters = Map(
+ "option" -> toDSOption("decompression"),
+ "config" -> toSQLConf(SQLConf.HADOOP_LINE_RECORD_READER_ENABLED.key)))
+ }
+
def cannotReadZipEntry(entry: String, path: String): SparkRuntimeException =
{
new SparkRuntimeException(
errorClass = "CANNOT_READ_ZIP_ENTRY",
diff --git
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/HadoopLineRecordReader.java
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/HadoopLineRecordReader.java
index 9a015cc740e4..b834a9b093b6 100644
---
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/HadoopLineRecordReader.java
+++
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/HadoopLineRecordReader.java
@@ -141,7 +141,7 @@ public class HadoopLineRecordReader extends
RecordReader<LongWritable, Text> {
// Try Spark's ZSTD decompression support. This is not
available in Hadoop's
// version of LineRecordReader.
Option<InputStream> decompressedStreamOpt =
- HadoopCodecStreams.createZstdInputStream(file,
fileIn);
+ HadoopCodecStreams.createZstdInputStream(job,
file, fileIn);
if (decompressedStreamOpt.isEmpty()) {
// File is either not ZSTD compressed or ZSTD codec is
not available.
throw e;
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
index 04dc7efaf98a..6cc8922e0de0 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormat.scala
@@ -25,10 +25,11 @@ import org.apache.hadoop.mapreduce.Job
import org.apache.spark.SparkContext
import org.apache.spark.paths.SparkPath
import org.apache.spark.sql.SparkSession
-import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow}
import org.apache.spark.sql.catalyst.expressions._
import
org.apache.spark.sql.catalyst.expressions.codegen.GenerateUnsafeProjection
import org.apache.spark.sql.catalyst.types.DataTypeUtils.toAttributes
+import org.apache.spark.sql.catalyst.util.CompressionCodecs
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.metric.SQLMetric
import org.apache.spark.sql.internal.{SessionStateHelper, SQLConf}
@@ -456,14 +457,41 @@ object FileFormat {
abstract class TextBasedFileFormat extends FileFormat with SessionStateHelper {
private var codecFactory: CompressionCodecFactory = _
+ /**
+ * Forces the `decompression` read option, when set, onto `hadoopConf`
before the format's
+ * [[buildReader]] broadcasts it, so the readers decompress every file with
that codec instead of
+ * inferring it from the file name extension
+ * (see [[CompressionCodecs.setReadCodecConfiguration]]).
+ */
+ override def buildReaderWithPartitionValues(
+ sparkSession: SparkSession,
+ dataSchema: StructType,
+ partitionSchema: StructType,
+ requiredSchema: StructType,
+ filters: Seq[Filter],
+ options: Map[String, String],
+ hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = {
+ CompressionCodecs.setReadCodecConfiguration(
+ hadoopConf, new FileSourceOptions(options).decompressionCodec)
+ super.buildReaderWithPartitionValues(
+ sparkSession, dataSchema, partitionSchema, requiredSchema, filters,
options, hadoopConf)
+ }
+
+ /**
+ * Whether the file at `path` can be split by its compression codec. When
the `decompression`
+ * read option is set, the splittability of that codec wins over the file
name extension, so
+ * the split decision always agrees with how
[[buildReaderWithPartitionValues]] decompresses.
+ */
override def isSplitable(
sparkSession: SparkSession,
options: Map[String, String],
path: Path): Boolean = {
- if (codecFactory == null) {
- codecFactory = new CompressionCodecFactory(getHadoopConf(sparkSession,
options))
+ new FileSourceOptions(options).decompressionCodecSplittable.getOrElse {
+ if (codecFactory == null) {
+ codecFactory = new CompressionCodecFactory(getHadoopConf(sparkSession,
options))
+ }
+ val codec = codecFactory.getCodec(path)
+ codec == null || codec.isInstanceOf[SplittableCompressionCodec]
}
- val codec = codecFactory.getCodec(path)
- codec == null || codec.isInstanceOf[SplittableCompressionCodec]
}
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
index dbfcff3f1fa5..745c8543fe5e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala
@@ -34,12 +34,13 @@ import org.apache.commons.compress.archivers.zip.ZipFile
import org.apache.commons.io.ByteOrderMark
import org.apache.commons.io.input.{BOMInputStream, CloseShieldInputStream}
import org.apache.hadoop.conf.Configuration
-import org.apache.hadoop.fs.{FSDataInputStream, GlobPattern, Path}
+import org.apache.hadoop.fs.{FileStatus, FSDataInputStream, GlobPattern, Path}
import org.apache.hadoop.io.Text
import org.apache.hadoop.util.LineReader
import org.apache.spark.{SparkEnv, TaskContext}
import org.apache.spark.internal.Logging
+import org.apache.spark.io.HadoopCodecStreams
import org.apache.spark.paths.SparkPath
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.errors.QueryExecutionErrors
@@ -208,6 +209,33 @@ object SupportsArchiveFormat {
name.endsWith(".zip") || name.endsWith(".7z")
}
+ /**
+ * Fails when the `decompression` read option is set for the archive at
`path`. The option is
+ * not supported for archives (yet): a container is always decoded based on
its file name
+ * extension, so fail rather than silently ignore the option. Called while
planning the scan so
+ * the error is raised on the driver and is not swallowed by
`ignoreCorruptFiles`.
+ */
+ def checkNoDecompressionCodec(path: Path, decompressionCodec:
Option[String]): Unit = {
+ if (decompressionCodec.isDefined) {
+ throw
QueryExecutionErrors.decompressionCodecForArchiveUnsupportedError(path.toString)
+ }
+ }
+
+ /**
+ * [[checkNoDecompressionCodec]] for every archive among `inputPaths`.
Called by the
+ * archive-aware schema inference paths on the driver, before their
inference tasks open any
+ * archive, so inference fails with the same error as the scan instead of a
task failure that
+ * `ignoreCorruptFiles` could swallow.
+ */
+ def checkNoDecompressionCodecForArchives(
+ inputPaths: Seq[FileStatus],
+ decompressionCodec: Option[String]): Unit = {
+ if (decompressionCodec.isDefined) {
+ inputPaths.iterator.map(_.getPath).find(isArchivePath)
+ .foreach(checkNoDecompressionCodec(_, decompressionCodec))
+ }
+ }
+
/** An archive's entries as lazy `(entry, stream)` pairs; closing releases
the container. */
private type ArchiveEntries = Iterator[(ArchiveEntry, InputStream)] with
Closeable
@@ -216,6 +244,8 @@ object SupportsArchiveFormat {
* `(entry, stream)` pairs.
*/
private def openArchiveStream(path: Path, conf: Configuration):
ArchiveEntries = {
+ // Backstop for any path that opens an archive without one of the
driver-side checks above.
+ checkNoDecompressionCodec(path,
Option(conf.get(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF)))
val name = path.getName.toLowerCase(Locale.ROOT)
name match {
case n if n.endsWith(".tar") || n.endsWith(".tar.gz") ||
n.endsWith(".tgz") =>
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
index d2075a3304df..8701a54f6152 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala
@@ -38,6 +38,7 @@ import org.apache.spark.rdd.{BinaryFileRDD, RDD}
import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow}
import org.apache.spark.sql.catalyst.csv.{CSVHeaderChecker, CSVInferSchema,
CSVOptions, UnivocityParser}
+import org.apache.spark.sql.catalyst.util.CompressionCodecs
import org.apache.spark.sql.classic.ClassicConversions.castToImpl
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.SQLExecution
@@ -170,6 +171,8 @@ abstract class CSVDataSource extends Serializable with
Logging with SupportsArch
sparkSession: SparkSession,
inputPaths: Seq[FileStatus],
parsedOptions: CSVOptions): StructType = {
+ SupportsArchiveFormat.checkNoDecompressionCodecForArchives(
+ inputPaths, parsedOptions.decompressionCodec)
val baseRdd = CSVDataSource.createBaseRdd(sparkSession, inputPaths,
parsedOptions)
// Inference must see the same entries the scan reads, so it honors
archivePathFilter too.
// Capture the glob string: the compiled GlobPattern is not serializable,
so each task
@@ -245,6 +248,9 @@ object CSVDataSource extends Logging {
options.parameters))
FileInputFormat.setInputPaths(job, paths: _*)
val conf = job.getConfiguration
+ // Honor the `decompression` option during multiLine schema inference too,
so the codec is
+ // forced rather than inferred from the file name extension.
+ CompressionCodecs.setReadCodecConfiguration(conf,
options.decompressionCodec)
val rdd = new BinaryFileRDD(
sparkSession.sparkContext,
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
index 388030dcf42d..6938ecdfbf15 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVFileFormat.scala
@@ -47,6 +47,7 @@ case class CSVFileFormat() extends TextBasedFileFormat with
DataSourceRegister {
// A tar archive is decompressed/unpacked as a sequential stream, so it
must be read as a
// single split rather than carved into byte ranges.
if (parsedOptions.archiveFormatEnabled &&
SupportsArchiveFormat.isArchivePath(path)) {
+ SupportsArchiveFormat.checkNoDecompressionCodec(path,
parsedOptions.decompressionCodec)
return false
}
CSVDataSource(parsedOptions).isSplitable &&
super.isSplitable(sparkSession, options, path)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
index b39c9acc33da..f204d56e4e56 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonDataSource.scala
@@ -37,7 +37,7 @@ import org.apache.spark.rdd.{BinaryFileRDD, RDD}
import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow}
import org.apache.spark.sql.catalyst.json.{CreateJacksonParser, JacksonParser,
JsonInferSchema, JSONOptions}
-import org.apache.spark.sql.catalyst.util.FailureSafeParser
+import org.apache.spark.sql.catalyst.util.{CompressionCodecs,
FailureSafeParser}
import org.apache.spark.sql.classic.ClassicConversions.castToImpl
import org.apache.spark.sql.errors.QueryExecutionErrors
import org.apache.spark.sql.execution.SQLExecution
@@ -148,6 +148,9 @@ object JsonDataSource {
val job =
Job.getInstance(sparkSession.sessionState.newHadoopConfWithOptions(
parsedOptions.parameters))
val conf = job.getConfiguration
+ // Honor the `decompression` option during multiLine schema inference too,
so the codec is
+ // forced rather than inferred from the file name extension.
+ CompressionCodecs.setReadCodecConfiguration(conf,
parsedOptions.decompressionCodec)
val name = paths.mkString(",")
FileInputFormat.setInputPaths(job, paths: _*)
new BinaryFileRDD(
@@ -289,6 +292,8 @@ object MultiLineJsonDataSource extends JsonDataSource {
sparkSession: SparkSession,
inputPaths: Seq[FileStatus],
parsedOptions: JSONOptions): StructType = {
+ SupportsArchiveFormat.checkNoDecompressionCodecForArchives(
+ inputPaths, parsedOptions.decompressionCodec)
val baseRdd = JsonDataSource.createBaseRdd(sparkSession, inputPaths,
parsedOptions)
// Inference must see the same entries the scan reads, so it honors
archivePathFilter too.
// Capture the glob string: the compiled GlobPattern is not serializable,
so each task
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
index eeeb96d0e6e5..9747bd688fbf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/json/JsonFileFormat.scala
@@ -41,6 +41,7 @@ case class JsonFileFormat() extends TextBasedFileFormat with
DataSourceRegister
path: Path): Boolean = {
val parsedOptions = getJsonOptions(sparkSession, options)
if (parsedOptions.archiveFormatEnabled &&
SupportsArchiveFormat.isArchivePath(path)) {
+ SupportsArchiveFormat.checkNoDecompressionCodec(path,
parsedOptions.decompressionCodec)
// A tar archive is read as one sequential stream (entry by entry), so
it is never split.
return false
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala
index 62d2cd4a2edd..0d346c99a4da 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala
@@ -64,6 +64,7 @@ case class TextFileFormat()
path: Path): Boolean = {
val textOptions = new TextOptions(options)
if (textOptions.archiveFormatEnabled &&
SupportsArchiveFormat.isArchivePath(path)) {
+ SupportsArchiveFormat.checkNoDecompressionCodec(path,
textOptions.decompressionCodec)
// A tar archive is read as one sequential stream (entry by entry), so
it is never split.
return false
}
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala
index 468d58974ed9..7bc1cef15b6e 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextOptions.scala
@@ -61,6 +61,7 @@ class TextOptions(@transient private val parameters:
CaseInsensitiveMap[String])
private[sql] object TextOptions extends DataSourceOptions {
val COMPRESSION = newOption("compression")
+ val DECOMPRESSION = newOption(DataSourceOptions.DECOMPRESSION)
val WHOLETEXT = newOption("wholetext")
val ENCODING = newOption("encoding")
val LINE_SEP = newOption("lineSep")
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TextBasedFileScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TextBasedFileScan.scala
index e11adb186c84..82a12ca282f1 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TextBasedFileScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/TextBasedFileScan.scala
@@ -18,10 +18,13 @@ package org.apache.spark.sql.execution.datasources.v2
import scala.jdk.CollectionConverters._
+import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.compress.CompressionCodecFactory
import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.FileSourceOptions
+import org.apache.spark.sql.catalyst.util.CompressionCodecs
import org.apache.spark.sql.util.CaseInsensitiveStringMap
import org.apache.spark.util.Utils
@@ -31,7 +34,31 @@ abstract class TextBasedFileScan(
@transient private lazy val codecFactory: CompressionCodecFactory = new
CompressionCodecFactory(
sparkSession.sessionState.newHadoopConfWithOptions(options.asScala.toMap))
- override def isSplitable(path: Path): Boolean = Utils.isFileSplittable(path,
codecFactory)
+ // The splittability of the `decompression` read option's codec, or `None`
when it is absent.
+ @transient private lazy val decompressionCodecSplittable: Option[Boolean] =
+ new FileSourceOptions(options.asScala.toMap).decompressionCodecSplittable
+
+ /**
+ * Whether the file at `path` can be split by its compression codec. When
the `decompression`
+ * read option is set, the splittability of that codec wins over the file
name extension, so
+ * the split decision always agrees with how the readers decompress (see
[[newReadHadoopConf]]).
+ */
+ override def isSplitable(path: Path): Boolean =
+ decompressionCodecSplittable.getOrElse(Utils.isFileSplittable(path,
codecFactory))
+
+ /**
+ * Creates the Hadoop configuration the readers use, from the scan options
(Hadoop configurations
+ * are case sensitive), with the `decompression` read option, when set,
forced onto it so
+ * the readers decompress every file with that codec instead of inferring it
from the file name
+ * extension (see [[CompressionCodecs.setReadCodecConfiguration]]).
+ */
+ protected def newReadHadoopConf(): Configuration = {
+ val hadoopConf = sparkSession.sessionState.newHadoopConfWithOptions(
+ options.asCaseSensitiveMap.asScala.toMap)
+ CompressionCodecs.setReadCodecConfiguration(
+ hadoopConf, new
FileSourceOptions(options.asScala.toMap).decompressionCodec)
+ hadoopConf
+ }
override def getFileUnSplittableReason(path: Path): String = {
assert(!isSplitable(path))
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala
index da14ead0f546..5ad58312dcbb 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/csv/CSVScan.scala
@@ -80,9 +80,7 @@ case class CSVScan(
val actualFilters =
pushedFilters.filterNot(_.references.contains(parsedOptions.columnNameOfCorruptRecord))
- val caseSensitiveMap = options.asCaseSensitiveMap.asScala.toMap
- // Hadoop Configurations are case sensitive.
- val hadoopConf =
sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap)
+ val hadoopConf = newReadHadoopConf()
val broadcastedConf =
SerializableConfiguration.broadcast(sparkSession.sparkContext,
hadoopConf)
// The partition values are already truncated in `FileScan.partitions`.
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala
index 9b19c6b433d7..993262bac02b 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScan.scala
@@ -73,9 +73,7 @@ case class JsonScan(
readDataSchema.head.name == parsedOptions.columnNameOfCorruptRecord) {
throw
QueryCompilationErrors.queryFromRawFilesIncludeCorruptRecordColumnError()
}
- val caseSensitiveMap = options.asCaseSensitiveMap.asScala.toMap
- // Hadoop Configurations are case sensitive.
- val hadoopConf =
sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap)
+ val hadoopConf = newReadHadoopConf()
val broadcastedConf =
SerializableConfiguration.broadcast(sparkSession.sparkContext,
hadoopConf)
// The partition values are already truncated in `FileScan.partitions`.
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextScan.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextScan.scala
index a351eb368c56..35464a4f3d25 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextScan.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/text/TextScan.scala
@@ -66,11 +66,7 @@ case class TextScan(
override def createReaderFactory(): PartitionReaderFactory = {
verifyReadSchema(readDataSchema)
- val hadoopConf = {
- val caseSensitiveMap = options.asCaseSensitiveMap.asScala.toMap
- // Hadoop Configurations are case sensitive.
- sparkSession.sessionState.newHadoopConfWithOptions(caseSensitiveMap)
- }
+ val hadoopConf = newReadHadoopConf()
val broadcastedConf =
SerializableConfiguration.broadcast(sparkSession.sparkContext,
hadoopConf)
TextPartitionReaderFactory(conf, broadcastedConf, readDataSchema,
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala
index 4ec07656f3f4..a0ac8d2f8b62 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlDataSource.scala
@@ -35,7 +35,7 @@ import org.apache.spark.internal.Logging
import org.apache.spark.rdd.{BinaryFileRDD, RDD}
import org.apache.spark.sql.{Dataset, Encoders, SparkSession}
import org.apache.spark.sql.catalyst.{FileSourceOptions, InternalRow}
-import org.apache.spark.sql.catalyst.util.FailureSafeParser
+import org.apache.spark.sql.catalyst.util.{CompressionCodecs,
FailureSafeParser}
import org.apache.spark.sql.catalyst.xml.{StaxXmlParser, StaxXMLRecordReader,
XmlInferSchema, XmlOptions}
import org.apache.spark.sql.classic.ClassicConversions.castToImpl
import org.apache.spark.sql.execution.SQLExecution
@@ -127,6 +127,9 @@ abstract class XmlDataSource extends Serializable with
Logging with SupportsArch
options.parameters))
FileInputFormat.setInputPaths(job, paths: _*)
val conf = job.getConfiguration
+ // Honor the `decompression` option during multiLine schema inference too,
so the codec is
+ // forced rather than inferred from the file name extension.
+ CompressionCodecs.setReadCodecConfiguration(conf,
options.decompressionCodec)
val rdd = new BinaryFileRDD(
sparkSession.sparkContext,
@@ -371,6 +374,8 @@ object MultiLineXmlDataSource extends XmlDataSource {
sparkSession: SparkSession,
inputPaths: Seq[FileStatus],
parsedOptions: XmlOptions): StructType = {
+ SupportsArchiveFormat.checkNoDecompressionCodecForArchives(
+ inputPaths, parsedOptions.decompressionCodec)
val baseRdd = createBaseRdd(sparkSession, inputPaths, parsedOptions)
// Inference must see the same entries the scan reads, so it honors
archivePathFilter too.
// Capture the glob string: the compiled GlobPattern is not serializable,
so each task
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
index 7823efd6ae81..d1736e50a5b4 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XmlFileFormat.scala
@@ -52,6 +52,7 @@ case class XmlFileFormat() extends TextBasedFileFormat with
DataSourceRegister {
path: Path): Boolean = {
val xmlOptions = getXmlOptions(sparkSession, options)
if (xmlOptions.archiveFormatEnabled &&
SupportsArchiveFormat.isArchivePath(path)) {
+ SupportsArchiveFormat.checkNoDecompressionCodec(path,
xmlOptions.decompressionCodec)
// A tar archive is read as one sequential stream (entry by entry), so
it is never split.
return false
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ArchiveReadSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ArchiveReadSuiteBase.scala
index 01a2521a737b..fcac5bf7ccf9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ArchiveReadSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/ArchiveReadSuiteBase.scala
@@ -19,10 +19,11 @@ package org.apache.spark.sql.execution.datasources
import java.io.File
import java.nio.file.Files
+import java.util.regex.Pattern
import org.apache.hadoop.fs.Path
-import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.{SparkConf, SparkException,
SparkUnsupportedOperationException}
import org.apache.spark.sql.{DataFrame, QueryTest, Row}
import org.apache.spark.sql.functions.{col, input_file_name, struct}
import org.apache.spark.sql.internal.SQLConf
@@ -168,6 +169,17 @@ trait ArchiveReadSuiteBase extends QueryTest with
SharedSparkSession {
*/
protected def supportsSchemaMerge: Boolean = true
+ /**
+ * Whether this format accepts the `decompression` read option. It does not
mean archives
+ * support the option: a format that accepts it must reject it for archive
inputs, failing with
+ * `UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE`, since an archive
is always decoded based
+ * on its file name extension. Gates the shared tests that assert that
rejection. CSV, JSON and
+ * XML set it true; formats that ignore the option (e.g. Parquet, ORC, Avro)
keep the default,
+ * because their archive reads succeed with it set. Text has its own archive
base,
+ * [[TextArchiveReadBase]], with the same test.
+ */
+ protected def supportsDecompressionCodec: Boolean = false
+
/**
* Whether default (non-merge) inference samples a single file rather than
scanning every input.
* Parquet samples one part-file (SPARK-11500), so a corrupt/missing
first-sampled archive fails
@@ -748,4 +760,66 @@ trait ArchiveReadSuiteBase extends QueryTest with
SharedSparkSession {
schema = complexReadSchema)
}
}
+
+ // ----- shared `decompression` tests (run when
`supportsDecompressionCodec`) -----
+
+ if (supportsDecompressionCodec) {
+ Seq(true, false).foreach { ignoreCorrupt =>
+ test("decompression is rejected for an archive " +
+ s"(ignoreCorruptFiles=$ignoreCorrupt)") {
+ archiveExtensions.foreach { ext =>
+ withArchiveFile(ext) { archive =>
+ writeArchive(archive, Seq(entryName(0) -> encodeFile(sampleDf((1,
"Alice")))))
+ // Raised while planning the scan, so `ignoreCorruptFiles` cannot
skip the archive.
+ withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key ->
ignoreCorrupt.toString) {
+ checkErrorMatchPVals(
+ exception = intercept[SparkUnsupportedOperationException] {
+ read(archive.getCanonicalPath, Map("decompression" ->
"gzip")).collect()
+ },
+ condition =
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE",
+ parameters = Map(
+ "option" -> "\"decompression\"",
+ "path" -> s".*${Pattern.quote(archive.getName)}"))
+ }
+ }
+ }
+ }
+ }
+
+ test("decompression is rejected for an archive among loose files") {
+ withTempDir { dir =>
+ writeArchive(
+ new File(dir, s"data.${archiveExtensions.head}"),
+ Seq(entryName(0) -> encodeFile(sampleDf((1, "Alice")))))
+ Files.write(new File(dir, entryName(1)).toPath,
encodeFile(sampleDf((2, "Bob"))))
+ val e = intercept[SparkUnsupportedOperationException] {
+ read(dir.getCanonicalPath, Map("decompression" -> "none")).collect()
+ }
+ assert(e.getCondition ==
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE")
+ }
+ }
+
+ if (supportsSchemaInference) {
+ Seq(true, false).foreach { ignoreCorrupt =>
+ test("decompression is rejected for an archive during schema inference
" +
+ s"(ignoreCorruptFiles=$ignoreCorrupt)") {
+ withArchiveFile() { archive =>
+ writeArchive(archive, Seq(entryName(0) -> encodeFile(sampleDf((1,
"Alice")))))
+ // Raised on the driver before any inference task opens the
archive, so it is neither
+ // wrapped in a task failure nor skipped by `ignoreCorruptFiles`.
+ withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key ->
ignoreCorrupt.toString) {
+ checkErrorMatchPVals(
+ exception = intercept[SparkUnsupportedOperationException] {
+ inferredSchema(Seq(archive.getCanonicalPath),
Map("decompression" -> "gzip"))
+ },
+ condition =
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE",
+ parameters = Map(
+ "option" -> "\"decompression\"",
+ "path" -> s".*${Pattern.quote(archive.getName)}"))
+ }
+ }
+ }
+ }
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
index 3665f857edd6..d0176259ea67 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CSVArchiveReadBase.scala
@@ -43,6 +43,8 @@ trait CSVArchiveReadBase extends ArchiveReadSuiteBase {
override protected def readSchema: String = "id INT, name STRING"
+ override protected def supportsDecompressionCodec: Boolean = true
+
// CSV infers its schema from row content (supportsSchemaInference defaults
true); inference is
// triggered by the `inferSchema` option. CSV is positional/header-keyed and
cannot represent
// nested types, so it opts out of the schema-merge and complex-type tests.
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
index 854fc55faed4..71a963666705 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/JSONArchiveReadBase.scala
@@ -44,6 +44,8 @@ trait JSONArchiveReadBase extends ArchiveReadSuiteBase {
override protected def readSchema: String = "id INT, name STRING"
+ override protected def supportsDecompressionCodec: Boolean = true
+
/** Raw JSON bytes, for tests that need precise control over the record
layout. */
protected def jsonBytes(s: String): Array[Byte] =
s.getBytes(StandardCharsets.UTF_8)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
index d9eae971e975..fe140bb7ae41 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormatSuite.scala
@@ -31,8 +31,10 @@ import
org.apache.commons.compress.archivers.tar.{TarArchiveEntry, TarArchiveOut
import org.apache.commons.compress.archivers.zip.{ZipArchiveEntry,
ZipArchiveOutputStream}
import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs.{GlobPattern, Path}
+import org.apache.hadoop.io.compress.GzipCodec
-import org.apache.spark.{SparkFunSuite, SparkRuntimeException, TaskContext,
TaskContextImpl}
+import org.apache.spark.{SparkFunSuite, SparkRuntimeException,
SparkUnsupportedOperationException, TaskContext, TaskContextImpl}
+import org.apache.spark.io.HadoopCodecStreams
import org.apache.spark.sql.catalyst.FileSourceOptions
/**
@@ -679,4 +681,33 @@ class SupportsArchiveFormatSuite extends SparkFunSuite {
assert(options.archivePathFilter.isEmpty)
assert(options.archivePathFilterPattern.isEmpty)
}
+
+ // ----- decompression
-------------------------------------------------------
+
+ test("readArchiveEntries: a forced decompression codec is rejected when the
archive is opened") {
+ // The driver-side checks normally reject the `decompression` option
first; this backstop in
+ // the archive reader covers any path that reaches it with the internal
codec key still set,
+ // both for a codec class and for `none` (encoded as an empty value).
+ withTempDir { dir =>
+ val entries = Seq(textEntry("a.csv", "1\n"))
+ val archives = Seq(
+ new File(dir, "data.tar") -> writeTar _,
+ new File(dir, "data.tar.gz") -> writeTarGz _,
+ new File(dir, "data.zip") -> writeZip _,
+ new File(dir, "data.7z") -> writeSevenZ _)
+ archives.foreach { case (file, write) => write(file, entries) }
+ for ((file, _) <- archives; codec <- Seq(classOf[GzipCodec].getName,
"")) {
+ val conf = new Configuration()
+ conf.set(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF, codec)
+ val e = intercept[SparkUnsupportedOperationException] {
+ SupportsArchiveFormat.readArchiveEntries(
+ new Path(file.toURI), conf, archivePathFilter = None) { (entry,
_) =>
+ Iterator.single(entry.getName)
+ }.toList
+ }
+ assert(e.getCondition ==
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE",
+ s"${file.getName} with codec '$codec'")
+ }
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
index 03c8b95d23c8..48f011ee0825 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/TextArchiveReadBase.scala
@@ -20,8 +20,9 @@ package org.apache.spark.sql.execution.datasources
import java.io.File
import java.nio.charset.StandardCharsets
import java.nio.file.Files
+import java.util.regex.Pattern
-import org.apache.spark.{SparkConf, SparkException}
+import org.apache.spark.{SparkConf, SparkException,
SparkUnsupportedOperationException}
import org.apache.spark.sql.{DataFrame, QueryTest, Row}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
@@ -93,6 +94,27 @@ trait TextArchiveReadBase extends QueryTest with
SharedSparkSession {
}
}
+ Seq(true, false).foreach { ignoreCorrupt =>
+ test(s"decompression is rejected for an archive
(ignoreCorruptFiles=$ignoreCorrupt)") {
+ archiveExtensions.foreach { ext =>
+ withArchiveFile(ext) { archive =>
+ writeArchive(archive, Seq("a.txt" -> textBytes("line1\n")))
+ // Raised while planning the scan, so `ignoreCorruptFiles` cannot
skip the archive.
+ withSQLConf(SQLConf.IGNORE_CORRUPT_FILES.key ->
ignoreCorrupt.toString) {
+ checkErrorMatchPVals(
+ exception = intercept[SparkUnsupportedOperationException] {
+ read(archive.getCanonicalPath, Map("decompression" ->
"gzip")).collect()
+ },
+ condition =
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_FOR_ARCHIVE",
+ parameters = Map(
+ "option" -> "\"decompression\"",
+ "path" -> s".*${Pattern.quote(archive.getName)}"))
+ }
+ }
+ }
+ }
+ }
+
test("an empty archive yields no rows") {
withArchiveFile() { archive =>
writeArchive(archive, Seq.empty)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
index ffa5484b3194..ed69689caf76 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/XMLArchiveReadBase.scala
@@ -44,6 +44,8 @@ trait XMLArchiveReadBase extends ArchiveReadSuiteBase {
override protected def readSchema: String = "id INT, name STRING"
+ override protected def supportsDecompressionCodec: Boolean = true
+
// XML infers from record content, unions fields across inputs by name, and
represents nested
// elements as structs, so it keeps all three `supports*` defaults
(inference, schema-merge,
// complex types) and runs the full shared test set. Inference needs no
trigger option, so
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
index b34b13efc512..ae07f97b89b9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
@@ -37,13 +37,14 @@ import
org.apache.hadoop.io.compress.{CompressionCodecFactory, GzipCodec}
import org.apache.logging.log4j.Level
import org.apache.spark._
-import org.apache.spark.io.ZStdCompressionCodec
+import org.apache.spark.io.{HadoopCodecStreams, ZStdCompressionCodec}
import org.apache.spark.sql._
import org.apache.spark.sql.catalyst.csv.CSVOptions
import org.apache.spark.sql.catalyst.expressions.ToStringBase
import org.apache.spark.sql.catalyst.util.{CharsetProvider, DateTimeTestUtils,
DateTimeUtils, HadoopCompressionCodec}
import org.apache.spark.sql.errors.QueryExecutionErrors.toSQLId
import org.apache.spark.sql.execution.datasources.CommonFileDataSourceSuite
+import
org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation,
FileScan}
import org.apache.spark.sql.internal.{LegacyBehaviorPolicy, SQLConf}
import org.apache.spark.sql.internal.SQLConf.BinaryOutputStyle
import org.apache.spark.sql.test.SharedSparkSession
@@ -3451,7 +3452,7 @@ abstract class CSVSuite
}
test("validate CSV Options") {
- assert(CSVOptions.getAllOptions.size == 44)
+ assert(CSVOptions.getAllOptions.size == 45)
// Please add validation on any new CSV options here
assert(CSVOptions.isValidOption("header"))
assert(CSVOptions.isValidOption("inferSchema"))
@@ -3490,6 +3491,7 @@ abstract class CSVSuite
assert(CSVOptions.isValidOption("encoding"))
assert(CSVOptions.isValidOption("charset"))
assert(CSVOptions.isValidOption("compression"))
+ assert(CSVOptions.isValidOption("decompression"))
assert(CSVOptions.isValidOption("codec"))
assert(CSVOptions.isValidOption("sep"))
assert(CSVOptions.isValidOption("extension"))
@@ -4303,6 +4305,195 @@ abstract class CSVSuite
}
}
}
+
+ // Writes `data` to `file` compressed with `codec` (a short name; "" means
no compression), using
+ // Spark's built-in Zstandard codec for zstd (which Hadoop may not be
compiled with) and Hadoop's
+ // CompressionCodecFactory otherwise. The file name intentionally does NOT
carry the codec's
+ // extension, so extension-based detection cannot pick the right codec and
the
+ // `decompression` option is what makes the read succeed.
+ private def writeCompressed(file: File, data: String, codec: String): Unit =
{
+ val codecFactory = new
CompressionCodecFactory(spark.sessionState.newHadoopConf())
+ val os = new FileOutputStream(file)
+ val out = codec match {
+ case "" => os
+ case "zstd" | "zstandard" | "zst" =>
+ new ZStdCompressionCodec(sparkConf).compressedOutputStream(os)
+ case name => codecFactory.getCodecByName(name).createOutputStream(os)
+ }
+ out.write(data.getBytes(StandardCharsets.UTF_8))
+ out.close()
+ }
+
+ test("SPARK-60217 decompression option overrides file name extension") {
+ val numRecord = 100
+ val records = (1 to numRecord).map(i => (i, s"value_$i"))
+ // Every read-supported codec, including zstd/zstandard/zst which the
writers do not support.
+ val codecs = HadoopCompressionCodec.values().map(_.lowerCaseName())
+ .filterNot(c => c == HadoopCompressionCodec.NONE.lowerCaseName() ||
+ c == HadoopCompressionCodec.UNCOMPRESSED.lowerCaseName()) ++
Seq("zstd", "zstandard", "zst")
+ for (multiLine <- Seq(true, false); codec <- codecs) {
+ withTempDir { dir =>
+ val data = ("id,value" +: records.map { case (id, v) => s"$id,$v"
}).mkString("\n")
+ // Misleading `.csv` name (no codec suffix): extension-based detection
would read it as
+ // plain text and fail. The `decompression` option forces the correct
codec.
+ val file = new File(dir, "data.csv")
+ writeCompressed(file, data, codec)
+
+ val df = spark.read
+ .option("decompression", codec)
+ .option("header", "true")
+ .option("inferSchema", "true")
+ .option("multiLine", multiLine.toString)
+ .csv(file.getCanonicalPath)
+ checkAnswer(df, records.map { case (id, v) => Row(id, v) })
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option decides whether a file is split") {
+ val records = (1 to 2000).map(i => (i, s"value_$i"))
+ val data = records.map { case (id, v) => s"$id,$v" }.mkString("\n")
+ val bzip2 = HadoopCompressionCodec.BZIP2.lowerCaseName()
+ val gzip = HadoopCompressionCodec.GZIP.lowerCaseName()
+ // (codec the file is written with, file name, `decompression`, multiLine,
whether it is split)
+ Seq(
+ // bzip2 is splittable even though the `.csv` name says nothing about it.
+ (bzip2, "data.csv", bzip2, false, true),
+ // bzip2 is splittable, but multiLine still reads each file as a single
split.
+ (bzip2, "data.csv", bzip2, true, false),
+ // gzip is not splittable, so a forced gzip `.csv` file is read as a
single split.
+ (gzip, "data.csv", gzip, false, false),
+ // `none` reads a plain file named `.gz` as-is, so it is split like any
plain file.
+ ("", "data.csv.gz", HadoopCompressionCodec.NONE.lowerCaseName(), false,
true)
+ ).foreach { case (writeCodec, name, readCodec, multiLine, split) =>
+ withTempDir { dir =>
+ val file = new File(dir, name)
+ writeCompressed(file, data, writeCodec)
+ withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "256") {
+ val df = spark.read
+ .option("decompression", readCodec)
+ .option("multiLine", multiLine.toString)
+ .schema("id INT, value STRING")
+ .csv(file.getCanonicalPath)
+ val numPartitions = df.rdd.getNumPartitions
+ assert(if (split) numPartitions > 1 else numPartitions == 1,
+ s"decompression=$readCodec on $name, multiLine=$multiLine:
$numPartitions partitions")
+ checkAnswer(df, records.map { case (id, v) => Row(id, v) })
+ }
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option accepts codec class names and any
letter case") {
+ withTempDir { dir =>
+ val file = new File(dir, "data.csv")
+ writeCompressed(file, "1,a\n2,b",
HadoopCompressionCodec.GZIP.lowerCaseName())
+ Seq(classOf[GzipCodec].getName, "GZIP", "GZip").foreach { codec =>
+ val df = spark.read
+ .option("decompression", codec)
+ .schema("id INT, value STRING")
+ .csv(file.getCanonicalPath)
+ checkAnswer(df, Seq(Row(1, "a"), Row(2, "b")))
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option in table options applies to table
reads") {
+ withTempDir { dir =>
+ // A gzip-compressed file whose `.csv` name says nothing about its
compression.
+ writeCompressed(
+ new File(dir, "data.csv"), "1,a\n2,b",
HadoopCompressionCodec.GZIP.lowerCaseName())
+ withTable("t") {
+ sql(
+ s"""CREATE TABLE t (id INT, value STRING) USING csv
+ |OPTIONS (path '${dir.toURI}', decompression
'gzip')""".stripMargin)
+ checkAnswer(sql("SELECT * FROM t"), Seq(Row(1, "a"), Row(2, "b")))
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option applies to streaming reads") {
+ withTempDir { dir =>
+ // A gzip-compressed file whose `.csv` name says nothing about its
compression.
+ writeCompressed(
+ new File(dir, "data.csv"), "1,a\n2,b",
HadoopCompressionCodec.GZIP.lowerCaseName())
+ val query = spark.readStream
+ .option("decompression", HadoopCompressionCodec.GZIP.lowerCaseName())
+ .schema("id INT, value STRING")
+ .csv(dir.getCanonicalPath)
+ .writeStream
+ .format("memory")
+ .queryName("decompression_codec_stream")
+ .start()
+ try {
+ query.processAllAvailable()
+ checkAnswer(
+ spark.table("decompression_codec_stream"), Seq(Row(1, "a"), Row(2,
"b")))
+ } finally {
+ query.stop()
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option none reads a file with a codec
extension as-is") {
+ withTempDir { dir =>
+ // A plain-text CSV that happens to carry a `.gz` extension.
Extension-based detection would
+ // try (and fail) to gunzip it; `decompression` = none must read the raw
bytes.
+ val file = new File(dir, "plain.csv.gz")
+ writeCompressed(file, "id,value\n1,a\n2,b", "")
+
+ val df = spark.read
+ .option("decompression", HadoopCompressionCodec.NONE.lowerCaseName())
+ .option("header", "true")
+ .csv(file.getCanonicalPath)
+ checkAnswer(df, Seq(Row("1", "a"), Row("2", "b")))
+ }
+ }
+
+ test("SPARK-60217 decompression option rejects an invalid codec") {
+ checkErrorMatchPVals(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.read.option("decompression",
"illegal").csv(testFile(carsFile)).collect()
+ },
+ condition = "CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION",
+ parameters = Map("codecName" -> "illegal", "availableCodecs" ->
".*zstd.*")
+ )
+ }
+
+ test("SPARK-60217 compression option does not force a codec on read") {
+ withTempDir { dir =>
+ // Unlike `decompression`, the write-side `compression` option (which
may be persisted
+ // in a table's options) must not force a codec on read: a plain file is
still read as-is.
+ val file = new File(dir, "plain.csv")
+ writeCompressed(file, "id,value\n1,a\n2,b", "")
+
+ val df = spark.read
+ .option("compression", HadoopCompressionCodec.GZIP.lowerCaseName())
+ .option("header", "true")
+ .csv(file.getCanonicalPath)
+ checkAnswer(df, Seq(Row("1", "a"), Row("2", "b")))
+ }
+ }
+
+ test("SPARK-60217 an inherited internal decompression codec key is not
honored") {
+ withTempDir { dir =>
+ val file = new File(dir, "plain.csv")
+ writeCompressed(file, "id,value\n1,a\n2,b", "")
+ // Session confs are copied into the read's Hadoop conf; the read and
the schema inference
+ // must unset the internal key since they did not set `decompression`,
so the plain file is
+ // not gunzipped. multiLine inference reads the files through its own
code path.
+ withSQLConf(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF ->
classOf[GzipCodec].getName) {
+ Seq(false, true).foreach { multiLine =>
+ val df = spark.read
+ .option("header", "true")
+ .option("inferSchema", "true")
+ .option("multiLine", multiLine.toString)
+ .csv(file.getCanonicalPath)
+ checkAnswer(df, Seq(Row(1, "a"), Row(2, "b")))
+ }
+ }
+ }
+ }
}
class CSVv1Suite extends CSVSuite {
@@ -4342,6 +4533,29 @@ class CSVv2Suite extends CSVSuite {
.sparkConf
.set(SQLConf.USE_V1_SOURCE_LIST, "")
+ test("SPARK-60217 unsplittable reason reports a forced decompression codec")
{
+ withTempDir { dir =>
+ val file = new File(dir, "data.csv")
+ Files.write(file.toPath, "1,a".getBytes(StandardCharsets.UTF_8))
+ def reason(options: Map[String, String]): String = {
+ // A user schema skips inference, so the file is never actually
decompressed.
+ val df = spark.read.options(options).schema("id INT, value STRING")
+ .csv(file.getCanonicalPath)
+ val scan = df.queryExecution.optimizedPlan.collectFirst {
+ case r: DataSourceV2ScanRelation => r.scan.asInstanceOf[FileScan]
+ }.get
+ scan.getFileUnSplittableReason(new Path(file.getCanonicalPath))
+ }
+ // The `.csv` name alone is splittable, so the forced codec is what
prevents splitting.
+ assert(reason(Map("decompression" ->
HadoopCompressionCodec.GZIP.lowerCaseName())) ==
+ "the file is compressed by unsplittable compression codec")
+ // A forced `none` keeps the file splittable, so multiLine is the reason.
+ assert(reason(Map(
+ "decompression" -> HadoopCompressionCodec.NONE.lowerCaseName(),
+ "multiLine" -> "true")) == "the csv datasource is set multiLine mode")
+ }
+ }
+
test("test for FAILFAST parsing mode on CSV v2") {
Seq(false, true).foreach { multiLine =>
checkError(
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
index e6e7d7eac0b8..a090b42a8657 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
@@ -49,6 +49,7 @@ import org.apache.spark.sql.catalyst.util.TypeUtils.toSQLType
import org.apache.spark.sql.errors.QueryExecutionErrors.toSQLId
import org.apache.spark.sql.execution.ExternalRDD
import org.apache.spark.sql.execution.datasources.{CommonFileDataSourceSuite,
DataSource, InMemoryFileIndex, NoopCache, PartitionedFile}
+import
org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation,
FileScan}
import org.apache.spark.sql.execution.datasources.v2.json.JsonScanBuilder
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
@@ -4465,7 +4466,7 @@ abstract class JsonSuite
}
test("SPARK-40667: validate JSON Options") {
- assert(JSONOptions.getAllOptions.size == 34)
+ assert(JSONOptions.getAllOptions.size == 35)
// Please add validation on any new Json options here
assert(JSONOptions.isValidOption("samplingRatio"))
assert(JSONOptions.isValidOption("primitivesAsString"))
@@ -4478,6 +4479,7 @@ abstract class JsonSuite
assert(JSONOptions.isValidOption("allowBackslashEscapingAnyCharacter"))
assert(JSONOptions.isValidOption("allowUnquotedControlChars"))
assert(JSONOptions.isValidOption("compression"))
+ assert(JSONOptions.isValidOption("decompression"))
assert(JSONOptions.isValidOption("mode"))
assert(JSONOptions.isValidOption("dropFieldIfAllNull"))
assert(JSONOptions.isValidOption("ignoreNullFields"))
@@ -4906,6 +4908,124 @@ abstract class JsonSuite
}
}
}
+
+ // Writes `data` to `file` compressed with `codec` (a short name; "" means
no compression), using
+ // Spark's built-in Zstandard codec for zstd (which Hadoop may not be
compiled with) and Hadoop's
+ // CompressionCodecFactory otherwise. The file name intentionally does NOT
carry the codec's
+ // extension, so extension-based detection cannot pick the right codec and
the
+ // `decompression` option is what makes the read succeed.
+ private def writeCompressed(file: File, data: String, codec: String): Unit =
{
+ val codecFactory = new
CompressionCodecFactory(spark.sessionState.newHadoopConf())
+ val os = new FileOutputStream(file)
+ val out = codec match {
+ case "" => os
+ case "zstd" | "zstandard" | "zst" =>
+ new ZStdCompressionCodec(sparkConf).compressedOutputStream(os)
+ case name => codecFactory.getCodecByName(name).createOutputStream(os)
+ }
+ out.write(data.getBytes(StandardCharsets.UTF_8))
+ out.close()
+ }
+
+ test("SPARK-60217 decompression option overrides file name extension") {
+ val numRecord = 100
+ val records = (1 to numRecord).map(i => (i, s"value_$i"))
+ // Every read-supported codec, including zstd/zstandard/zst which the
writers do not support.
+ val codecs = HadoopCompressionCodec.values().map(_.lowerCaseName())
+ .filterNot(c => c == HadoopCompressionCodec.NONE.lowerCaseName() ||
+ c == HadoopCompressionCodec.UNCOMPRESSED.lowerCaseName()) ++
Seq("zstd", "zstandard", "zst")
+ for (multiLine <- Seq(true, false); codec <- codecs) {
+ withTempDir { dir =>
+ val lines = records.map { case (id, v) => s"""{"id": $id, "value":
"$v"}""" }
+ val data = if (multiLine) lines.mkString("[\n", ",\n", "\n]") else
lines.mkString("\n")
+ // Misleading `.json` name (no codec suffix): extension-based
detection would read it as
+ // plain text and fail. The `decompression` option forces the correct
codec.
+ val file = new File(dir, "data.json")
+ writeCompressed(file, data, codec)
+
+ val df = spark.read
+ .option("decompression", codec)
+ .option("multiLine", multiLine.toString)
+ .json(file.getCanonicalPath)
+ checkAnswer(df.select("id", "value"), records.map { case (id, v) =>
Row(id, v) })
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option none reads a file with a codec
extension as-is") {
+ withTempDir { dir =>
+ // A plain-text JSON that happens to carry a `.gz` extension.
Extension-based detection would
+ // try (and fail) to gunzip it; `decompression` = none must read the raw
bytes.
+ val file = new File(dir, "plain.json.gz")
+ writeCompressed(file, """{"id": 1, "value": "a"}""", "")
+
+ val df = spark.read
+ .option("decompression", HadoopCompressionCodec.NONE.lowerCaseName())
+ .json(file.getCanonicalPath)
+ checkAnswer(df.select("id", "value"), Seq(Row(1, "a")))
+ }
+ }
+
+ test("SPARK-60217 decompression option decides whether a file is split") {
+ val records = (1 to 2000).map(i => (i, s"value_$i"))
+ val lines = records.map { case (id, v) => s"""{"id": $id, "value":
"$v"}""" }
+ val bzip2 = HadoopCompressionCodec.BZIP2.lowerCaseName()
+ // (codec the file is written with, file name, `decompression`, multiLine,
whether it is split)
+ Seq(
+ // bzip2 is splittable even though the `.json` name says nothing about
it.
+ (bzip2, "data.json", bzip2, false, true),
+ // bzip2 is splittable, but multiLine still reads each file as a single
split.
+ (bzip2, "data.json", bzip2, true, false),
+ // gzip is not splittable, so a forced gzip `.json` file is read as a
single split.
+ (GZIP.lowerCaseName(), "data.json", GZIP.lowerCaseName(), false, false),
+ // `none` reads a plain file named `.gz` as-is, so it is split like any
plain file.
+ ("", "data.json.gz", HadoopCompressionCodec.NONE.lowerCaseName(), false,
true)
+ ).foreach { case (writeCodec, name, readCodec, multiLine, split) =>
+ withTempDir { dir =>
+ val file = new File(dir, name)
+ val data = if (multiLine) lines.mkString("[\n", ",\n", "\n]") else
lines.mkString("\n")
+ writeCompressed(file, data, writeCodec)
+ withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "256") {
+ val df = spark.read
+ .option("decompression", readCodec)
+ .option("multiLine", multiLine.toString)
+ .schema("id INT, value STRING")
+ .json(file.getCanonicalPath)
+ val numPartitions = df.rdd.getNumPartitions
+ assert(if (split) numPartitions > 1 else numPartitions == 1,
+ s"decompression=$readCodec on $name, multiLine=$multiLine:
$numPartitions partitions")
+ checkAnswer(df, records.map { case (id, v) => Row(id, v) })
+ }
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option rejects an invalid codec") {
+ withTempPath { dir =>
+ Seq("""{"a": 1}""").toDF().write.text(dir.getCanonicalPath)
+ checkErrorMatchPVals(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.read.option("decompression",
"illegal").json(dir.getCanonicalPath).collect()
+ },
+ condition = "CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION",
+ parameters = Map("codecName" -> "illegal", "availableCodecs" ->
".*zstd.*")
+ )
+ }
+ }
+
+ test("SPARK-60217 compression option does not force a codec on read") {
+ withTempDir { dir =>
+ // Unlike `decompression`, the write-side `compression` option (which
may be persisted
+ // in a table's options) must not force a codec on read: a plain file is
still read as-is.
+ val file = new File(dir, "plain.json")
+ writeCompressed(file, """{"a": 1}""", "")
+
+ val df = spark.read
+ .option("compression", GZIP.lowerCaseName())
+ .json(file.getCanonicalPath)
+ checkAnswer(df, Row(1L) :: Nil)
+ }
+ }
}
class JsonV1Suite extends JsonSuite {
@@ -4950,6 +5070,28 @@ class JsonV2Suite extends JsonSuite {
}
}
}
+
+ test("SPARK-60217 unsplittable reason reports a forced decompression codec")
{
+ withTempDir { dir =>
+ val file = new File(dir, "data.json")
+ Files.write(file.toPath, """{"id":
1}""".getBytes(StandardCharsets.UTF_8))
+ def reason(options: Map[String, String]): String = {
+ // A user schema skips inference, so the file is never actually
decompressed.
+ val df = spark.read.options(options).schema("id
INT").json(file.getCanonicalPath)
+ val scan = df.queryExecution.optimizedPlan.collectFirst {
+ case r: DataSourceV2ScanRelation => r.scan.asInstanceOf[FileScan]
+ }.get
+ scan.getFileUnSplittableReason(new Path(file.getCanonicalPath))
+ }
+ // The `.json` name alone is splittable, so the forced codec is what
prevents splitting.
+ assert(reason(Map("decompression" -> GZIP.lowerCaseName())) ==
+ "the file is compressed by unsplittable compression codec")
+ // A forced `none` keeps the file splittable, so multiLine is the reason.
+ assert(reason(Map(
+ "decompression" -> HadoopCompressionCodec.NONE.lowerCaseName(),
+ "multiLine" -> "true")) == "the json datasource is set multiLine mode")
+ }
+ }
}
class JsonLegacyTimeParserSuite extends JsonSuite {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
index 38fea013a246..de1394cb6993 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/text/TextSuite.scala
@@ -17,19 +17,21 @@
package org.apache.spark.sql.execution.datasources.text
-import java.io.{File, FileOutputStream}
+import java.io.{File, FileOutputStream, InputStream}
import java.nio.charset.StandardCharsets
import java.nio.file.Files
+import org.apache.hadoop.fs.Path
import org.apache.hadoop.io.SequenceFile.CompressionType
-import org.apache.hadoop.io.compress.{CompressionCodecFactory, GzipCodec}
+import org.apache.hadoop.io.compress.{CompressionCodecFactory,
CompressionInputStream, Decompressor, DefaultCodec, GzipCodec}
-import org.apache.spark.{SparkConf, SparkIllegalArgumentException, TestUtils}
-import org.apache.spark.io.ZStdCompressionCodec
+import org.apache.spark.{SparkConf, SparkIllegalArgumentException,
SparkUnsupportedOperationException, TestUtils}
+import org.apache.spark.io.{HadoopCodecStreams, ZStdCompressionCodec}
import org.apache.spark.sql.{AnalysisException, DataFrame, Row, SaveMode}
import org.apache.spark.sql.catalyst.util.HadoopCompressionCodec
-import org.apache.spark.sql.catalyst.util.HadoopCompressionCodec.{BZIP2,
DEFLATE, GZIP, LZ4, NONE, SNAPPY}
+import org.apache.spark.sql.catalyst.util.HadoopCompressionCodec.{BZIP2,
DEFLATE, GZIP, LZ4, NONE, SNAPPY, UNCOMPRESSED}
import org.apache.spark.sql.execution.datasources.CommonFileDataSourceSuite
+import
org.apache.spark.sql.execution.datasources.v2.{DataSourceV2ScanRelation,
FileScan}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{StringType, StructType}
@@ -249,9 +251,10 @@ abstract class TextSuite extends SharedSparkSession with
CommonFileDataSourceSui
}
test("SPARK-40667: validate Text Options") {
- assert(TextOptions.getAllOptions.size == 4)
+ assert(TextOptions.getAllOptions.size == 5)
// Please add validation on any new Text options here
assert(TextOptions.isValidOption("compression"))
+ assert(TextOptions.isValidOption("decompression"))
assert(TextOptions.isValidOption("wholetext"))
assert(TextOptions.isValidOption("encoding"))
assert(TextOptions.isValidOption("lineSep"))
@@ -341,6 +344,189 @@ abstract class TextSuite extends SharedSparkSession with
CommonFileDataSourceSui
}
}
}
+
+ // Writes `data` to `file` compressed with `codec` (a short name; "" means
no compression), using
+ // Spark's built-in Zstandard codec for zstd (which Hadoop may not be
compiled with) and Hadoop's
+ // CompressionCodecFactory otherwise. The file name intentionally does NOT
carry the codec's
+ // extension, so extension-based detection cannot pick the right codec and
the
+ // `decompression` option is what makes the read succeed.
+ private def writeCompressed(file: File, data: String, codec: String): Unit =
{
+ val codecFactory = new
CompressionCodecFactory(spark.sessionState.newHadoopConf())
+ val os = new FileOutputStream(file)
+ val out = codec match {
+ case "" => os
+ case "zstd" | "zstandard" | "zst" =>
+ new ZStdCompressionCodec(sparkConf).compressedOutputStream(os)
+ case name => codecFactory.getCodecByName(name).createOutputStream(os)
+ }
+ out.write(data.getBytes(StandardCharsets.UTF_8))
+ out.close()
+ }
+
+ test("SPARK-60217 decompression option overrides file name extension") {
+ val numRecord = 100
+ val data = (1 to numRecord).map(i => s"$i, value_$i")
+ // Every read-supported codec, including zstd/zstandard/zst which the
writers do not support.
+ val codecs = HadoopCompressionCodec.values().map(_.lowerCaseName())
+ .filterNot(c => c == NONE.lowerCaseName() || c ==
UNCOMPRESSED.lowerCaseName()) ++
+ Seq("zstd", "zstandard", "zst")
+ for (wholeText <- Seq(true, false); codec <- codecs) {
+ withTempDir { dir =>
+ // Misleading `.txt` name (no codec suffix): extension-based detection
would read it as
+ // plain text and fail. The `decompression` option forces the correct
codec.
+ val file = new File(dir, "data.txt")
+ writeCompressed(file, data.mkString("\n"), codec)
+
+ val df = spark.read
+ .option("decompression", codec)
+ .option("wholeText", wholeText.toString)
+ .text(file.getCanonicalPath)
+ val expected = if (wholeText) Seq(Row(data.mkString("\n"))) else
data.map(Row(_))
+ checkAnswer(df, expected)
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option none reads a file with a codec
extension as-is") {
+ withTempDir { dir =>
+ val data = "not really gzip compressed"
+ // A plain-text file that happens to carry a `.gz` extension.
Extension-based detection would
+ // try (and fail) to gunzip it; `decompression` = none must read the raw
bytes.
+ val file = new File(dir, "plain.txt.gz")
+ writeCompressed(file, data, "")
+
+ Seq(NONE, UNCOMPRESSED).foreach { codec =>
+ val df = spark.read
+ .option("decompression", codec.lowerCaseName())
+ .text(file.getCanonicalPath)
+ checkAnswer(df, Row(data) :: Nil)
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option rejects an invalid codec") {
+ withTempPath { dir =>
+ Seq("value").toDF().write.text(dir.getCanonicalPath)
+ // The suggested codec list is built from a Map, so its order is not
asserted here; the read
+ // set additionally includes zstd/zstandard/zst on top of the
write-supported codecs.
+ checkErrorMatchPVals(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.read.option("decompression",
"illegal").text(dir.getCanonicalPath).collect()
+ },
+ condition = "CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION",
+ parameters = Map(
+ "codecName" -> "illegal",
+ "availableCodecs" -> ".*zstd.*")
+ )
+ }
+ }
+
+ test("SPARK-60217 decompression option decides whether a file is split") {
+ val lines = (1 to 2000).map(i => s"line_$i")
+ // (codec the file is written with, file name, `decompression`, whether it
is split)
+ Seq(
+ // bzip2 is splittable even though the `.txt` name says nothing about it.
+ (BZIP2.lowerCaseName(), "data.txt", BZIP2.lowerCaseName(), true),
+ // gzip is not splittable, so a forced gzip `.txt` file is read as a
single split.
+ (GZIP.lowerCaseName(), "data.txt", GZIP.lowerCaseName(), false),
+ // `none` reads a plain file named `.gz` as-is, so it is split like any
plain file.
+ ("", "data.txt.gz", NONE.lowerCaseName(), true)
+ ).foreach { case (writeCodec, name, readCodec, split) =>
+ withTempDir { dir =>
+ val file = new File(dir, name)
+ writeCompressed(file, lines.mkString("\n"), writeCodec)
+ withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "256") {
+ val df = spark.read.option("decompression",
readCodec).text(file.getCanonicalPath)
+ val numPartitions = df.rdd.getNumPartitions
+ assert(if (split) numPartitions > 1 else numPartitions == 1,
+ s"decompression=$readCodec on $name: $numPartitions partitions")
+ checkAnswer(df, lines.map(Row(_)))
+ }
+ }
+ }
+ }
+
+ test("SPARK-60217 a forced codec that fails is not replaced by zstd based on
the extension") {
+ withTempDir { dir =>
+ // A zstd file with a `.zst` name, read with a forced codec whose
streams cannot be created.
+ // The zstd fallback for codecs that fail must not kick in, since a
codec was forced.
+ val file = new File(dir, "data.txt.zst")
+ writeCompressed(file, "a\nb", "zstd")
+ Seq(false, true).foreach { wholeText =>
+ val e = intercept[Exception] {
+ spark.read
+ .option("decompression",
classOf[FailingDecompressionCodec].getName)
+ .option("wholetext", wholeText.toString)
+ .text(file.getCanonicalPath).collect()
+ }
+ assert(Iterator.iterate[Throwable](e)(_.getCause).takeWhile(_ != null)
+ .exists(_.getMessage == FailingDecompressionCodec.message),
s"wholetext=$wholeText")
+ }
+ }
+ }
+
+ test("SPARK-60217 a forced codec that cannot be created does not leak the
file stream") {
+ withTempDir { dir =>
+ val file = new File(dir, "data.txt")
+ writeCompressed(file, "a\nb", "")
+ // The codec is only created in the read task, after the file is opened.
The suite checks
+ // after each test that every opened file stream was closed.
+ Seq(false, true).foreach { wholeText =>
+ val e = intercept[Exception] {
+ spark.read
+ .option("decompression",
classOf[UnconstructibleDecompressionCodec].getName)
+ .option("wholetext", wholeText.toString)
+ .text(file.getCanonicalPath).collect()
+ }
+ assert(Iterator.iterate[Throwable](e)(_.getCause).takeWhile(_ != null)
+ .exists(_.getMessage == UnconstructibleDecompressionCodec.message),
+ s"wholetext=$wholeText")
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option is rejected with the legacy line
reader") {
+ withTempPath { dir =>
+ Seq("value").toDF().write.text(dir.getCanonicalPath)
+ // Hadoop's LineRecordReader infers the codec from the extension and
would ignore the option.
+ withSQLConf(SQLConf.HADOOP_LINE_RECORD_READER_ENABLED.key -> "false") {
+ checkError(
+ exception = intercept[SparkUnsupportedOperationException] {
+ spark.read.option("decompression",
"gzip").text(dir.getCanonicalPath).collect()
+ },
+ condition =
"UNSUPPORTED_FEATURE.DECOMPRESSION_CODEC_WITH_LEGACY_LINE_READER",
+ parameters = Map(
+ "option" -> "\"decompression\"",
+ "config" ->
s"\"${SQLConf.HADOOP_LINE_RECORD_READER_ENABLED.key}\""))
+ }
+ }
+ }
+
+ test("SPARK-60217 an inherited internal decompression codec key is not
honored") {
+ withTempDir { dir =>
+ val file = new File(dir, "plain.txt")
+ writeCompressed(file, "plain text", "")
+ // Session confs are copied into the read's Hadoop conf; the read must
unset the internal key
+ // since it did not set `decompression`, so the plain file is not
gunzipped.
+ withSQLConf(HadoopCodecStreams.DECOMPRESSION_CODEC_CONF ->
classOf[GzipCodec].getName) {
+ checkAnswer(spark.read.text(file.getCanonicalPath), Row("plain text")
:: Nil)
+ }
+ }
+ }
+
+ test("SPARK-60217 compression option does not force a codec on read") {
+ withTempDir { dir =>
+ // Unlike `decompression`, the write-side `compression` option (which
may be persisted
+ // in a table's options) must not force a codec on read: a plain file is
still read as-is.
+ val file = new File(dir, "plain.txt")
+ writeCompressed(file, "plain text", "")
+
+ val df = spark.read
+ .option("compression", GZIP.lowerCaseName())
+ .text(file.getCanonicalPath)
+ checkAnswer(df, Row("plain text") :: Nil)
+ }
+ }
}
class TextV1Suite extends TextSuite {
@@ -356,4 +542,48 @@ class TextV2Suite extends TextSuite {
super
.sparkConf
.set(SQLConf.USE_V1_SOURCE_LIST, "")
+
+ test("SPARK-60217 unsplittable reason reports a forced decompression codec")
{
+ withTempDir { dir =>
+ val file = new File(dir, "data.txt")
+ Files.write(file.toPath, "line".getBytes(StandardCharsets.UTF_8))
+ def reason(options: Map[String, String]): String = {
+ val df = spark.read.options(options).text(file.getCanonicalPath)
+ val scan = df.queryExecution.optimizedPlan.collectFirst {
+ case r: DataSourceV2ScanRelation => r.scan.asInstanceOf[FileScan]
+ }.get
+ scan.getFileUnSplittableReason(new Path(file.getCanonicalPath))
+ }
+ // The `.txt` name alone is splittable, so the forced codec is what
prevents splitting.
+ assert(reason(Map("decompression" -> GZIP.lowerCaseName())) ==
+ "the file is compressed by unsplittable compression codec")
+ // A forced `none` keeps the file splittable, so wholetext is the reason.
+ assert(reason(Map("decompression" -> NONE.lowerCaseName(), "wholetext"
-> "true")) ==
+ "the text datasource is set wholetext mode")
+ }
+ }
+}
+
+/** A codec that fails to create any input stream, like a codec whose native
library is missing. */
+class FailingDecompressionCodec extends DefaultCodec {
+ override def createInputStream(in: InputStream): CompressionInputStream =
+ throw new UnsupportedOperationException(FailingDecompressionCodec.message)
+
+ override def createInputStream(
+ in: InputStream,
+ decompressor: Decompressor): CompressionInputStream =
+ throw new UnsupportedOperationException(FailingDecompressionCodec.message)
+}
+
+object FailingDecompressionCodec {
+ val message = "Cannot create a FailingDecompressionCodec input stream"
+}
+
+/** A codec that cannot be created, like one whose class fails to initialize
on an executor. */
+class UnconstructibleDecompressionCodec extends DefaultCodec {
+ throw new
UnsupportedOperationException(UnconstructibleDecompressionCodec.message)
+}
+
+object UnconstructibleDecompressionCodec {
+ val message = "Cannot create an UnconstructibleDecompressionCodec"
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlSuite.scala
index 3c2000214b98..7e92294fbd9c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/xml/XmlSuite.scala
@@ -37,7 +37,7 @@ import org.apache.hadoop.io.{LongWritable, Text}
import org.apache.hadoop.io.compress.{CompressionCodecFactory, GzipCodec}
import org.xml.sax.SAXException
-import org.apache.spark.{DebugFilesystem, SparkConf, SparkException}
+import org.apache.spark.{DebugFilesystem, SparkConf, SparkException,
SparkIllegalArgumentException}
import org.apache.spark.io.ZStdCompressionCodec
import org.apache.spark.sql.{AnalysisException, DataFrame, Dataset, Encoders,
Row, SaveMode, YearUDT}
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.UDTEncoder
@@ -3679,7 +3679,7 @@ class XmlSuite
}
test("validate XML Options") {
- assert(XmlOptions.getAllOptions.size == 32)
+ assert(XmlOptions.getAllOptions.size == 33)
// Please add validation on any new XML options here
assert(XmlOptions.isValidOption("rowTag"))
assert(XmlOptions.isValidOption("rootTag"))
@@ -3698,6 +3698,7 @@ class XmlSuite
assert(XmlOptions.isValidOption("mode"))
assert(XmlOptions.isValidOption("locale"))
assert(XmlOptions.isValidOption("compression"))
+ assert(XmlOptions.isValidOption("decompression"))
assert(XmlOptions.isValidOption("multiLine"))
assert(XmlOptions.isValidOption("samplingRatio"))
assert(XmlOptions.isValidOption("columnNameOfCorruptRecord"))
@@ -3783,6 +3784,97 @@ class XmlSuite
}
}
+ // Writes `data` to `file` compressed with `codec` (a short name; "" means
no compression), using
+ // Spark's built-in Zstandard codec for zstd (which Hadoop may not be
compiled with) and Hadoop's
+ // CompressionCodecFactory otherwise. The file name intentionally does NOT
carry the codec's
+ // extension, so extension-based detection cannot pick the right codec and
the
+ // `decompression` option is what makes the read succeed.
+ private def writeCompressed(file: File, data: String, codec: String): Unit =
{
+ val codecFactory = new
CompressionCodecFactory(spark.sessionState.newHadoopConf())
+ val os = new FileOutputStream(file)
+ val out = codec match {
+ case "" => os
+ case "zstd" | "zstandard" | "zst" =>
+ new ZStdCompressionCodec(sparkConf).compressedOutputStream(os)
+ case name => codecFactory.getCodecByName(name).createOutputStream(os)
+ }
+ out.write(data.getBytes(StandardCharsets.UTF_8))
+ out.close()
+ }
+
+ private def xmlDoc(records: Seq[(Int, String)]): String =
+ records.map { case (id, v) =>
+ s" <ROW><id>$id</id><value>$v</value></ROW>"
+ }.mkString("<ROWS>\n", "\n", "\n</ROWS>")
+
+ test("SPARK-60217 decompression option overrides file name extension") {
+ val numRecord = 100
+ val records = (1 to numRecord).map(i => (i, s"value_$i"))
+ // Every read-supported codec, including zstd/zstandard/zst which the
writers do not support.
+ val codecs = HadoopCompressionCodec.values().map(_.lowerCaseName())
+ .filterNot(c => c == HadoopCompressionCodec.NONE.lowerCaseName() ||
+ c == HadoopCompressionCodec.UNCOMPRESSED.lowerCaseName()) ++
Seq("zstd", "zstandard", "zst")
+ codecs.foreach { codec =>
+ withTempDir { dir =>
+ // Misleading `.xml` name (no codec suffix): extension-based detection
would read it as
+ // plain text and fail. The `decompression` option forces the correct
codec.
+ val file = new File(dir, "data.xml")
+ writeCompressed(file, xmlDoc(records), codec)
+
+ val df = spark.read
+ .option("decompression", codec)
+ .option("rowTag", "ROW")
+ .xml(file.getCanonicalPath)
+ checkAnswer(df.select("id", "value"), records.map { case (id, v) =>
Row(id, v) })
+ }
+ }
+ }
+
+ test("SPARK-60217 decompression option none reads a file with a codec
extension as-is") {
+ withTempDir { dir =>
+ // A plain-text XML that happens to carry a `.gz` extension.
Extension-based detection would
+ // try (and fail) to gunzip it; `decompression` = none must read the raw
bytes.
+ val file = new File(dir, "plain.xml.gz")
+ writeCompressed(file, xmlDoc(Seq((1, "a"))), "")
+
+ val df = spark.read
+ .option("decompression", HadoopCompressionCodec.NONE.lowerCaseName())
+ .option("rowTag", "ROW")
+ .xml(file.getCanonicalPath)
+ checkAnswer(df.select("id", "value"), Seq(Row(1, "a")))
+ }
+ }
+
+ test("SPARK-60217 decompression option rejects an invalid codec") {
+ withTempDir { dir =>
+ val file = new File(dir, "data.xml")
+ writeCompressed(file, xmlDoc(Seq((1, "a"))), "")
+ checkErrorMatchPVals(
+ exception = intercept[SparkIllegalArgumentException] {
+ spark.read.option("decompression", "illegal").option("rowTag", "ROW")
+ .xml(file.getCanonicalPath).collect()
+ },
+ condition = "CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION",
+ parameters = Map("codecName" -> "illegal", "availableCodecs" ->
".*zstd.*")
+ )
+ }
+ }
+
+ test("SPARK-60217 compression option does not force a codec on read") {
+ withTempDir { dir =>
+ // Unlike `decompression`, the write-side `compression` option (which
may be persisted
+ // in a table's options) must not force a codec on read: a plain file is
still read as-is.
+ val file = new File(dir, "plain.xml")
+ writeCompressed(file, xmlDoc(Seq((1, "a"))), "")
+
+ val df = spark.read
+ .option("compression", HadoopCompressionCodec.GZIP.lowerCaseName())
+ .option("rowTag", "ROW")
+ .xml(file.getCanonicalPath)
+ checkAnswer(df, Row(1L, "a") :: Nil)
+ }
+ }
+
}
class XmlSuiteWithLegacyParser extends XmlSuite {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]